2024-03-25T17:40:56.547 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T17:40:56.548 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621304 --verbose --name amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} --owner scheduled_amathuri@teuthology -- /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621304/orig.config.yaml 2024-03-25T17:40:56.590 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T17:40:56.674 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621304 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7621304' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6785 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi083.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOcxMrfnS1278Hh4wj+koiCBiAB8jKgK0bfQL+LG0aENfyqJ17ESta7jqnj6tNr7rTpGvziLwO1VCqfv+Nq7aD8= smithi103.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMS12b5fIyshgoPfOLpw9fDz704LM+m9FF/4ofwZhCYoyPVNHdYpMCoCpEbltvNc7n3eayx8YCzLHJb5tUCrrfs= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_13:28:33 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-25T17:40:56.674 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-25T17:40:56.675 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-25T17:40:56.675 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-25T17:40:56.676 INFO:teuthology.task.internal:Checking packages... 2024-03-25T17:40:56.699 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-25T17:40:56.699 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-25T17:40:56.700 INFO:teuthology.packaging:ref: None 2024-03-25T17:40:56.700 INFO:teuthology.packaging:tag: None 2024-03-25T17:40:56.700 INFO:teuthology.packaging:branch: main 2024-03-25T17:40:56.700 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T17:40:56.700 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-25T17:40:56.930 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2381.g44a9b520 2024-03-25T17:40:56.932 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-25T17:40:56.939 INFO:teuthology.task.internal:no buildpackages task found 2024-03-25T17:40:56.939 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-25T17:40:56.946 INFO:teuthology.task.internal:Saving configuration 2024-03-25T17:40:56.961 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-25T17:40:56.967 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T17:40:56.995 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi083.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621304', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 17:35:18.260584', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOcxMrfnS1278Hh4wj+koiCBiAB8jKgK0bfQL+LG0aENfyqJ17ESta7jqnj6tNr7rTpGvziLwO1VCqfv+Nq7aD8='} 2024-03-25T17:40:57.019 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi103.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621304', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 17:35:18.262282', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMS12b5fIyshgoPfOLpw9fDz704LM+m9FF/4ofwZhCYoyPVNHdYpMCoCpEbltvNc7n3eayx8YCzLHJb5tUCrrfs='} 2024-03-25T17:40:57.019 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-25T17:40:57.026 INFO:teuthology.task.internal:roles: ubuntu@smithi083.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-25T17:40:57.027 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-25T17:40:57.027 INFO:teuthology.run_tasks:Running task console_log... 2024-03-25T17:40:57.097 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f2ac29d90d0>, signals=[15]) 2024-03-25T17:40:57.097 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-25T17:40:57.103 INFO:teuthology.task.internal:Opening connections... 2024-03-25T17:40:57.104 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi083.front.sepia.ceph.com 2024-03-25T17:40:57.105 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:40:57.198 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi103.front.sepia.ceph.com 2024-03-25T17:40:57.199 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:40:57.285 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-25T17:40:57.293 DEBUG:teuthology.orchestra.run.smithi083:> uname -m 2024-03-25T17:40:57.310 INFO:teuthology.orchestra.run.smithi083.stdout:x86_64 2024-03-25T17:40:57.311 DEBUG:teuthology.orchestra.run.smithi083:> cat /etc/os-release 2024-03-25T17:40:57.367 INFO:teuthology.orchestra.run.smithi083.stdout:NAME="CentOS Stream" 2024-03-25T17:40:57.367 INFO:teuthology.orchestra.run.smithi083.stdout:VERSION="9" 2024-03-25T17:40:57.367 INFO:teuthology.orchestra.run.smithi083.stdout:ID="centos" 2024-03-25T17:40:57.367 INFO:teuthology.orchestra.run.smithi083.stdout:ID_LIKE="rhel fedora" 2024-03-25T17:40:57.367 INFO:teuthology.orchestra.run.smithi083.stdout:VERSION_ID="9" 2024-03-25T17:40:57.367 INFO:teuthology.orchestra.run.smithi083.stdout:PLATFORM_ID="platform:el9" 2024-03-25T17:40:57.367 INFO:teuthology.orchestra.run.smithi083.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T17:40:57.367 INFO:teuthology.orchestra.run.smithi083.stdout:ANSI_COLOR="0;31" 2024-03-25T17:40:57.367 INFO:teuthology.orchestra.run.smithi083.stdout:LOGO="fedora-logo-icon" 2024-03-25T17:40:57.367 INFO:teuthology.orchestra.run.smithi083.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T17:40:57.368 INFO:teuthology.orchestra.run.smithi083.stdout:HOME_URL="https://centos.org/" 2024-03-25T17:40:57.368 INFO:teuthology.orchestra.run.smithi083.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T17:40:57.368 INFO:teuthology.orchestra.run.smithi083.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T17:40:57.368 INFO:teuthology.orchestra.run.smithi083.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T17:40:57.368 INFO:teuthology.lock.ops:Updating smithi083.front.sepia.ceph.com on lock server 2024-03-25T17:40:57.389 DEBUG:teuthology.orchestra.run.smithi103:> uname -m 2024-03-25T17:40:57.407 INFO:teuthology.orchestra.run.smithi103.stdout:x86_64 2024-03-25T17:40:57.408 DEBUG:teuthology.orchestra.run.smithi103:> cat /etc/os-release 2024-03-25T17:40:57.464 INFO:teuthology.orchestra.run.smithi103.stdout:NAME="CentOS Stream" 2024-03-25T17:40:57.464 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION="9" 2024-03-25T17:40:57.464 INFO:teuthology.orchestra.run.smithi103.stdout:ID="centos" 2024-03-25T17:40:57.464 INFO:teuthology.orchestra.run.smithi103.stdout:ID_LIKE="rhel fedora" 2024-03-25T17:40:57.464 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION_ID="9" 2024-03-25T17:40:57.464 INFO:teuthology.orchestra.run.smithi103.stdout:PLATFORM_ID="platform:el9" 2024-03-25T17:40:57.464 INFO:teuthology.orchestra.run.smithi103.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T17:40:57.465 INFO:teuthology.orchestra.run.smithi103.stdout:ANSI_COLOR="0;31" 2024-03-25T17:40:57.465 INFO:teuthology.orchestra.run.smithi103.stdout:LOGO="fedora-logo-icon" 2024-03-25T17:40:57.465 INFO:teuthology.orchestra.run.smithi103.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T17:40:57.465 INFO:teuthology.orchestra.run.smithi103.stdout:HOME_URL="https://centos.org/" 2024-03-25T17:40:57.465 INFO:teuthology.orchestra.run.smithi103.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T17:40:57.465 INFO:teuthology.orchestra.run.smithi103.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T17:40:57.465 INFO:teuthology.orchestra.run.smithi103.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T17:40:57.465 INFO:teuthology.lock.ops:Updating smithi103.front.sepia.ceph.com on lock server 2024-03-25T17:40:57.489 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-25T17:40:57.498 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-25T17:40:57.503 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-25T17:40:57.504 DEBUG:teuthology.orchestra.run.smithi083:> test '!' -e /home/ubuntu/cephtest 2024-03-25T17:40:57.506 DEBUG:teuthology.orchestra.run.smithi103:> test '!' -e /home/ubuntu/cephtest 2024-03-25T17:40:57.522 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-25T17:40:57.528 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-25T17:40:57.528 DEBUG:teuthology.orchestra.run.smithi083:> test -z $(ls -A /var/lib/ceph) 2024-03-25T17:40:57.563 DEBUG:teuthology.orchestra.run.smithi103:> test -z $(ls -A /var/lib/ceph) 2024-03-25T17:40:57.683 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-25T17:40:57.740 INFO:teuthology.run_tasks:Running task kernel... 2024-03-25T17:40:57.755 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-25T17:40:57.755 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-25T17:40:57.755 DEBUG:teuthology.orchestra.run.smithi083:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T17:40:57.756 DEBUG:teuthology.orchestra.run.smithi103:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T17:40:57.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:40:57.772 DEBUG:teuthology.orchestra.run.smithi083:> uname -r 2024-03-25T17:40:57.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:40:57.773 DEBUG:teuthology.orchestra.run.smithi103:> uname -r 2024-03-25T17:40:57.829 INFO:teuthology.orchestra.run.smithi083.stdout:5.14.0-361.el9.x86_64 2024-03-25T17:40:57.830 INFO:teuthology.task.kernel:Running kernel on smithi083: 5.14.0-361.el9.x86_64 2024-03-25T17:40:57.830 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum install -y kernel 2024-03-25T17:40:57.831 INFO:teuthology.orchestra.run.smithi103.stdout:5.14.0-361.el9.x86_64 2024-03-25T17:40:57.831 INFO:teuthology.task.kernel:Running kernel on smithi103: 5.14.0-361.el9.x86_64 2024-03-25T17:40:57.831 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum install -y kernel 2024-03-25T17:41:00.160 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-25T17:41:00.619 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS-9 - AppStream 6.0 MB/s | 6.7 MB 00:01 2024-03-25T17:41:01.852 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS-9 - BaseOS 3.5 MB/s | 2.0 MB 00:00 2024-03-25T17:41:02.019 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS-9 - BaseOS 7.2 MB/s | 2.0 MB 00:00 2024-03-25T17:41:10.817 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - BaseOS 972 kB/s | 8.0 MB 00:08 2024-03-25T17:41:12.367 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - BaseOS 809 kB/s | 8.0 MB 00:10 2024-03-25T17:41:20.947 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - AppStream 2.1 MB/s | 19 MB 00:09 2024-03-25T17:41:23.072 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - AppStream 2.0 MB/s | 19 MB 00:09 2024-03-25T17:41:33.526 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - CRB 701 kB/s | 6.1 MB 00:08 2024-03-25T17:41:35.081 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 16 kB 00:00 2024-03-25T17:41:35.926 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - CRB 670 kB/s | 6.1 MB 00:09 2024-03-25T17:41:36.049 INFO:teuthology.orchestra.run.smithi083.stdout:Extra Packages for Enterprise Linux 24 MB/s | 21 MB 00:00 2024-03-25T17:41:37.356 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - Extras packages 55 kB/s | 16 kB 00:00 2024-03-25T17:41:38.357 INFO:teuthology.orchestra.run.smithi103.stdout:Extra Packages for Enterprise Linux 23 MB/s | 21 MB 00:00 2024-03-25T17:41:40.974 INFO:teuthology.orchestra.run.smithi083.stdout:lab-extras 24 kB/s | 1.7 kB 00:00 2024-03-25T17:41:43.114 INFO:teuthology.orchestra.run.smithi083.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T17:41:43.114 INFO:teuthology.orchestra.run.smithi083.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T17:41:43.205 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T17:41:43.216 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T17:41:43.216 INFO:teuthology.orchestra.run.smithi083.stdout: Package Architecture Version Repository Size 2024-03-25T17:41:43.216 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T17:41:43.216 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T17:41:43.216 INFO:teuthology.orchestra.run.smithi083.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-25T17:41:43.216 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-03-25T17:41:43.216 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-25T17:41:43.216 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-25T17:41:43.217 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-25T17:41:43.217 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:41:43.217 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T17:41:43.217 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T17:41:43.217 INFO:teuthology.orchestra.run.smithi083.stdout:Install 4 Packages 2024-03-25T17:41:43.217 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:41:43.218 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 76 M 2024-03-25T17:41:43.218 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 123 M 2024-03-25T17:41:43.218 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T17:41:43.340 INFO:teuthology.orchestra.run.smithi103.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-25T17:41:43.791 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:41:43.811 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:41:43.830 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:41:44.166 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:41:44.185 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:41:44.199 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:41:44.241 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:41:44.255 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:41:44.275 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:41:44.389 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:41:44.406 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:41:44.428 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:41:44.554 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:41:44.573 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:41:44.595 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:41:44.914 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:41:44.941 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:41:44.957 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:41:44.972 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:41:45.007 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:41:45.007 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:41:45.111 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:41:45.115 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:41:45.139 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:41:45.281 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:41:45.306 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:41:45.307 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:41:45.331 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:41:45.333 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:41:45.356 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:41:45.380 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:41:45.410 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:41:45.442 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:41:45.474 INFO:teuthology.orchestra.run.smithi103.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T17:41:45.474 INFO:teuthology.orchestra.run.smithi103.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T17:41:45.544 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:41:45.544 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:41:45.564 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-25T17:41:45.571 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T17:41:45.571 INFO:teuthology.orchestra.run.smithi103.stdout: Package Architecture Version Repository Size 2024-03-25T17:41:45.571 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T17:41:45.571 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-25T17:41:45.572 INFO:teuthology.orchestra.run.smithi103.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-25T17:41:45.572 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-03-25T17:41:45.572 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-25T17:41:45.572 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-25T17:41:45.572 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-25T17:41:45.572 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T17:41:45.572 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-25T17:41:45.572 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T17:41:45.572 INFO:teuthology.orchestra.run.smithi103.stdout:Install 4 Packages 2024-03-25T17:41:45.572 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T17:41:45.573 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 76 M 2024-03-25T17:41:45.573 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 123 M 2024-03-25T17:41:45.573 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-25T17:41:45.579 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:41:45.585 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:41:45.585 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:41:45.654 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:41:45.674 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:41:45.677 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:41:45.695 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:41:45.696 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:41:45.696 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:41:45.712 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:41:45.715 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:41:45.715 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:41:45.764 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:41:45.766 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:41:45.802 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:41:45.821 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:41:45.890 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:41:45.920 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:41:45.928 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T17:41:45.940 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:41:45.962 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T17:41:45.981 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T17:41:46.276 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T17:41:46.310 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:41:46.314 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T17:41:46.331 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:41:46.337 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T17:41:46.354 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:41:46.371 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:41:46.388 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:41:46.449 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:41:46.458 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:41:46.471 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:41:46.507 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:41:46.527 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:41:46.545 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:41:46.586 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:41:46.588 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:41:46.621 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:41:46.622 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:41:46.638 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:41:46.640 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:41:46.658 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:41:46.833 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:41:46.859 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:41:46.874 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:41:46.881 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:41:46.900 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:41:46.915 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:41:46.955 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:41:46.960 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:41:46.961 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:41:46.971 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:41:46.991 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:41:47.005 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:41:47.029 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:41:47.029 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:41:47.068 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:41:47.086 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:41:47.095 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:41:47.119 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:41:47.135 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:41:47.135 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:41:47.172 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:41:47.277 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:41:47.295 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:41:47.298 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:41:47.326 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:41:47.343 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:41:47.357 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:41:47.450 INFO:teuthology.orchestra.run.smithi083.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 16 kB/s | 66 kB 00:04 2024-03-25T17:41:47.532 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:41:47.583 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:41:47.584 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:41:47.606 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:41:47.641 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:41:47.641 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:41:47.658 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:41:47.660 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:41:47.695 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:41:47.713 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:41:47.715 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:41:47.764 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:41:47.823 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T17:41:47.880 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T17:41:47.898 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:41:47.928 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T17:41:48.175 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T17:41:48.238 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T17:41:48.283 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T17:41:48.383 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:41:48.435 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:41:48.461 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:41:48.517 INFO:teuthology.orchestra.run.smithi083.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 2.8 MB/s | 15 MB 00:05 2024-03-25T17:41:48.805 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:41:48.840 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:41:48.863 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:41:48.880 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:41:48.892 INFO:teuthology.orchestra.run.smithi083.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 5.9 MB/s | 33 MB 00:05 2024-03-25T17:41:48.894 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:41:48.899 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:41:48.916 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:41:48.951 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:41:48.993 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:41:49.025 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:41:49.085 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:41:49.166 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:41:49.194 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:41:49.217 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:41:49.231 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:41:49.352 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:41:49.366 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:41:49.438 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:41:49.442 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:41:49.442 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:41:49.477 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:41:49.478 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:41:49.542 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:41:49.578 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:41:49.578 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:41:49.583 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:41:49.594 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:41:49.595 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:41:49.657 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:41:49.670 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:41:49.671 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:41:49.747 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:41:49.804 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:41:49.837 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:41:49.841 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:41:49.846 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:41:49.891 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:41:49.902 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:41:49.911 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:41:49.920 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:41:49.920 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:41:49.928 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:41:49.976 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:41:49.979 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:41:50.035 INFO:teuthology.orchestra.run.smithi103.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 15 kB/s | 66 kB 00:04 2024-03-25T17:41:50.041 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:41:50.211 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:41:50.227 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:41:50.316 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:41:50.331 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:41:50.420 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:41:50.461 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:41:50.525 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:41:50.543 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:41:50.898 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T17:41:51.002 INFO:teuthology.orchestra.run.smithi103.stdout:(2/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 6.2 MB/s | 33 MB 00:05 2024-03-25T17:41:51.009 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:41:51.058 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T17:41:51.091 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:41:51.126 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:41:51.185 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:41:51.262 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:41:51.344 INFO:teuthology.orchestra.run.smithi103.stdout:(3/4): kernel-core-5.14.0-432.el9.x86_64.rpm 2.6 MB/s | 15 MB 00:05 2024-03-25T17:41:51.434 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:41:51.452 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:41:51.542 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:41:51.645 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:41:51.740 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:41:51.779 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:41:51.940 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:41:52.052 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:41:52.186 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:41:52.298 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:41:52.355 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:41:52.372 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:41:52.650 INFO:teuthology.orchestra.run.smithi083.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 5.3 MB/s | 27 MB 00:05 2024-03-25T17:41:52.655 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:41:52.655 INFO:teuthology.orchestra.run.smithi083.stdout:Total 8.0 MB/s | 76 MB 00:09 2024-03-25T17:41:52.729 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T17:41:52.886 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T17:41:53.253 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:41:53.322 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T17:41:53.421 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:41:53.509 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:41:53.545 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:41:53.693 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T17:41:53.693 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T17:41:53.710 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:41:53.859 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:41:53.893 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:41:53.976 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:41:54.053 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:41:54.134 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:41:54.188 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:41:54.699 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T17:41:54.699 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T17:41:55.119 INFO:teuthology.orchestra.run.smithi103.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 5.4 MB/s | 27 MB 00:05 2024-03-25T17:41:55.124 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:41:55.125 INFO:teuthology.orchestra.run.smithi103.stdout:Total 8.0 MB/s | 76 MB 00:09 2024-03-25T17:41:55.785 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-25T17:41:56.158 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-25T17:41:56.159 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-25T17:41:57.154 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-25T17:41:57.154 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-25T17:41:57.157 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T17:41:57.775 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-25T17:41:57.878 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:41:59.633 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-25T17:42:00.237 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-25T17:42:00.329 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:42:00.658 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:42:00.789 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:42:03.103 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:42:03.240 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:42:05.522 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:42:05.556 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:42:08.079 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:42:08.140 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:42:10.297 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:42:12.858 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:42:39.420 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:42:39.421 INFO:teuthology.orchestra.run.smithi083.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T17:42:39.421 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:42:39.466 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:42:39.837 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:42:39.837 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-25T17:42:39.838 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:42:39.838 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:42:40.250 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:42:40.250 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:42:40.250 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T17:42:40.250 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-25T17:42:40.250 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-25T17:42:40.250 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-25T17:42:40.250 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-25T17:42:40.250 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:42:40.251 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T17:42:40.601 DEBUG:teuthology.orchestra.run.smithi083:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T17:42:40.698 INFO:teuthology.orchestra.run.smithi083.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T17:42:40.698 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T17:42:40.698 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-25T17:42:40.698 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T17:42:40.698 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T17:42:40.699 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-25T17:42:40.699 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi083.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T17:42:40.699 DEBUG:teuthology.orchestra.run.smithi083:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T17:42:40.779 INFO:teuthology.orchestra.run.smithi083.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T17:42:40.779 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T17:42:40.779 DEBUG:teuthology.orchestra.run.smithi083:> sudo rpm -qi grub2-tools 2024-03-25T17:42:40.813 INFO:teuthology.orchestra.run.smithi083.stdout:Name : grub2-tools 2024-03-25T17:42:40.813 INFO:teuthology.orchestra.run.smithi083.stdout:Epoch : 1 2024-03-25T17:42:40.813 INFO:teuthology.orchestra.run.smithi083.stdout:Version : 2.06 2024-03-25T17:42:40.813 INFO:teuthology.orchestra.run.smithi083.stdout:Release : 68.el9 2024-03-25T17:42:40.813 INFO:teuthology.orchestra.run.smithi083.stdout:Architecture: x86_64 2024-03-25T17:42:40.813 INFO:teuthology.orchestra.run.smithi083.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T17:42:40.813 INFO:teuthology.orchestra.run.smithi083.stdout:Group : Unspecified 2024-03-25T17:42:40.814 INFO:teuthology.orchestra.run.smithi083.stdout:Size : 8264254 2024-03-25T17:42:40.814 INFO:teuthology.orchestra.run.smithi083.stdout:License : GPLv3+ 2024-03-25T17:42:40.814 INFO:teuthology.orchestra.run.smithi083.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T17:42:40.814 INFO:teuthology.orchestra.run.smithi083.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T17:42:40.814 INFO:teuthology.orchestra.run.smithi083.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T17:42:40.814 INFO:teuthology.orchestra.run.smithi083.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T17:42:40.814 INFO:teuthology.orchestra.run.smithi083.stdout:Packager : builder@centos.org 2024-03-25T17:42:40.814 INFO:teuthology.orchestra.run.smithi083.stdout:Vendor : CentOS 2024-03-25T17:42:40.814 INFO:teuthology.orchestra.run.smithi083.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T17:42:40.814 INFO:teuthology.orchestra.run.smithi083.stdout:Summary : Support tools for GRUB. 2024-03-25T17:42:40.814 INFO:teuthology.orchestra.run.smithi083.stdout:Description : 2024-03-25T17:42:40.814 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:42:40.814 INFO:teuthology.orchestra.run.smithi083.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T17:42:40.814 INFO:teuthology.orchestra.run.smithi083.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T17:42:40.815 INFO:teuthology.orchestra.run.smithi083.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T17:42:40.815 INFO:teuthology.orchestra.run.smithi083.stdout:hardware devices. 2024-03-25T17:42:40.815 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:42:40.815 INFO:teuthology.orchestra.run.smithi083.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T17:42:40.816 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T17:42:40.816 INFO:teuthology.task.kernel:Updating grub on smithi083 to boot 5.14.0-432.el9.x86_64 2024-03-25T17:42:40.816 DEBUG:teuthology.orchestra.run.smithi083:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T17:42:41.436 INFO:teuthology.orchestra.run.smithi083.stderr:Generating grub configuration file ... 2024-03-25T17:42:41.920 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:42:41.921 INFO:teuthology.orchestra.run.smithi103.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T17:42:41.921 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T17:42:41.965 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:42:42.292 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:42:42.292 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-25T17:42:42.292 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:42:42.292 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:42:42.743 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:42:42.743 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T17:42:42.743 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-25T17:42:42.743 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-25T17:42:42.743 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-25T17:42:42.743 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-25T17:42:42.743 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-25T17:42:42.744 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T17:42:42.744 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-25T17:42:42.865 INFO:teuthology.orchestra.run.smithi083.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T17:42:42.884 INFO:teuthology.orchestra.run.smithi083.stderr:done 2024-03-25T17:42:42.886 DEBUG:teuthology.orchestra.run.smithi083:> mktemp 2024-03-25T17:42:42.901 INFO:teuthology.orchestra.run.smithi083.stdout:/tmp/tmp.X9XBa0qb8C 2024-03-25T17:42:42.902 DEBUG:teuthology.orchestra.run.smithi083:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.X9XBa0qb8C 2024-03-25T17:42:42.976 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 0666 /tmp/tmp.X9XBa0qb8C 2024-03-25T17:42:43.055 DEBUG:teuthology.orchestra.run.smithi103:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T17:42:43.117 DEBUG:teuthology.orchestra.remote:smithi083:/tmp/tmp.X9XBa0qb8C is 6KB 2024-03-25T17:42:43.156 INFO:teuthology.orchestra.run.smithi103.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T17:42:43.156 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T17:42:43.157 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-25T17:42:43.157 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T17:42:43.157 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T17:42:43.157 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-25T17:42:43.157 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi103.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T17:42:43.157 DEBUG:teuthology.orchestra.run.smithi103:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T17:42:43.164 DEBUG:teuthology.orchestra.run.smithi083:> rm -fr /tmp/tmp.X9XBa0qb8C 2024-03-25T17:42:43.181 DEBUG:teuthology.orchestra.run.smithi083:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T17:42:43.237 INFO:teuthology.orchestra.run.smithi103.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T17:42:43.237 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T17:42:43.237 DEBUG:teuthology.orchestra.run.smithi103:> sudo rpm -qi grub2-tools 2024-03-25T17:42:43.242 INFO:teuthology.orchestra.run.smithi083.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T17:42:43.243 INFO:teuthology.orchestra.run.smithi083.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T17:42:43.243 INFO:teuthology.orchestra.run.smithi083.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T17:42:43.243 INFO:teuthology.orchestra.run.smithi083.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-25T17:42:43.244 DEBUG:teuthology.orchestra.run.smithi083:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-25T17:42:43.279 INFO:teuthology.orchestra.run.smithi103.stdout:Name : grub2-tools 2024-03-25T17:42:43.279 INFO:teuthology.orchestra.run.smithi103.stdout:Epoch : 1 2024-03-25T17:42:43.279 INFO:teuthology.orchestra.run.smithi103.stdout:Version : 2.06 2024-03-25T17:42:43.279 INFO:teuthology.orchestra.run.smithi103.stdout:Release : 68.el9 2024-03-25T17:42:43.279 INFO:teuthology.orchestra.run.smithi103.stdout:Architecture: x86_64 2024-03-25T17:42:43.280 INFO:teuthology.orchestra.run.smithi103.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T17:42:43.280 INFO:teuthology.orchestra.run.smithi103.stdout:Group : Unspecified 2024-03-25T17:42:43.280 INFO:teuthology.orchestra.run.smithi103.stdout:Size : 8264254 2024-03-25T17:42:43.280 INFO:teuthology.orchestra.run.smithi103.stdout:License : GPLv3+ 2024-03-25T17:42:43.280 INFO:teuthology.orchestra.run.smithi103.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T17:42:43.280 INFO:teuthology.orchestra.run.smithi103.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T17:42:43.280 INFO:teuthology.orchestra.run.smithi103.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T17:42:43.280 INFO:teuthology.orchestra.run.smithi103.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T17:42:43.280 INFO:teuthology.orchestra.run.smithi103.stdout:Packager : builder@centos.org 2024-03-25T17:42:43.280 INFO:teuthology.orchestra.run.smithi103.stdout:Vendor : CentOS 2024-03-25T17:42:43.280 INFO:teuthology.orchestra.run.smithi103.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T17:42:43.280 INFO:teuthology.orchestra.run.smithi103.stdout:Summary : Support tools for GRUB. 2024-03-25T17:42:43.280 INFO:teuthology.orchestra.run.smithi103.stdout:Description : 2024-03-25T17:42:43.280 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T17:42:43.280 INFO:teuthology.orchestra.run.smithi103.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T17:42:43.281 INFO:teuthology.orchestra.run.smithi103.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T17:42:43.281 INFO:teuthology.orchestra.run.smithi103.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T17:42:43.281 INFO:teuthology.orchestra.run.smithi103.stdout:hardware devices. 2024-03-25T17:42:43.281 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T17:42:43.281 INFO:teuthology.orchestra.run.smithi103.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T17:42:43.282 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T17:42:43.282 INFO:teuthology.task.kernel:Updating grub on smithi103 to boot 5.14.0-432.el9.x86_64 2024-03-25T17:42:43.282 DEBUG:teuthology.orchestra.run.smithi103:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T17:42:43.398 DEBUG:teuthology.orchestra.run.smithi083:> sudo shutdown -r now 2024-03-25T17:42:43.824 INFO:teuthology.orchestra.run.smithi103.stderr:Generating grub configuration file ... 2024-03-25T17:42:45.213 INFO:teuthology.orchestra.run.smithi103.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T17:42:45.247 INFO:teuthology.orchestra.run.smithi103.stderr:done 2024-03-25T17:42:45.249 DEBUG:teuthology.orchestra.run.smithi103:> mktemp 2024-03-25T17:42:45.265 INFO:teuthology.orchestra.run.smithi103.stdout:/tmp/tmp.gDc9ZvgtEn 2024-03-25T17:42:45.265 DEBUG:teuthology.orchestra.run.smithi103:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.gDc9ZvgtEn 2024-03-25T17:42:45.333 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0666 /tmp/tmp.gDc9ZvgtEn 2024-03-25T17:42:45.470 DEBUG:teuthology.orchestra.remote:smithi103:/tmp/tmp.gDc9ZvgtEn is 6KB 2024-03-25T17:42:45.518 DEBUG:teuthology.orchestra.run.smithi103:> rm -fr /tmp/tmp.gDc9ZvgtEn 2024-03-25T17:42:45.532 DEBUG:teuthology.orchestra.run.smithi103:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T17:42:45.597 INFO:teuthology.orchestra.run.smithi103.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T17:42:45.598 INFO:teuthology.orchestra.run.smithi103.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T17:42:45.598 INFO:teuthology.orchestra.run.smithi103.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T17:42:45.598 INFO:teuthology.orchestra.run.smithi103.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-25T17:42:45.599 DEBUG:teuthology.orchestra.run.smithi103:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-25T17:42:45.733 DEBUG:teuthology.orchestra.run.smithi103:> sudo shutdown -r now 2024-03-25T17:43:13.426 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-25T17:43:13.427 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-03-25T17:43:13.428 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:43:15.738 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-25T17:43:15.741 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-03-25T17:43:15.741 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:43:31.874 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.83 2024-03-25T17:43:40.882 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-03-25T17:43:40.883 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:43:43.934 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.83 2024-03-25T17:43:50.302 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-03-25T17:43:55.940 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-03-25T17:43:55.941 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:43:59.303 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-03-25T17:43:59.304 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:44:02.366 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-03-25T17:44:14.379 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-03-25T17:44:14.380 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:44:27.795 DEBUG:teuthology.orchestra.run.smithi083:> true 2024-03-25T17:44:28.195 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-03-25T17:44:28.196 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-432.el9.x86_64"... 2024-03-25T17:44:28.196 DEBUG:teuthology.orchestra.run.smithi083:> uname -r 2024-03-25T17:44:28.210 INFO:teuthology.orchestra.run.smithi083.stdout:5.14.0-432.el9.x86_64 2024-03-25T17:44:28.210 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-25T17:44:28.210 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T17:44:28.210 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T17:44:29.211 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-25T17:44:29.211 DEBUG:teuthology.orchestra.run.smithi083:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T17:44:29.281 INFO:teuthology.orchestra.run.smithi083.stdout:ttyS1 2024-03-25T17:44:29.324 DEBUG:teuthology.parallel:result is None 2024-03-25T17:44:32.798 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-03-25T17:44:47.800 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-03-25T17:44:47.800 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:44:48.276 DEBUG:teuthology.orchestra.run.smithi103:> true 2024-03-25T17:44:48.815 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-03-25T17:44:48.815 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-432.el9.x86_64"... 2024-03-25T17:44:48.815 DEBUG:teuthology.orchestra.run.smithi103:> uname -r 2024-03-25T17:44:48.829 INFO:teuthology.orchestra.run.smithi103.stdout:5.14.0-432.el9.x86_64 2024-03-25T17:44:48.829 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-25T17:44:48.829 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T17:44:48.830 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T17:44:49.830 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-25T17:44:49.830 DEBUG:teuthology.orchestra.run.smithi103:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T17:44:49.901 INFO:teuthology.orchestra.run.smithi103.stdout:ttyS1 2024-03-25T17:44:49.944 DEBUG:teuthology.parallel:result is None 2024-03-25T17:44:49.944 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-25T17:44:49.951 INFO:teuthology.task.internal:Creating test directory... 2024-03-25T17:44:49.951 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T17:44:49.958 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T17:44:49.973 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-25T17:44:49.979 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-25T17:44:49.989 INFO:teuthology.task.internal:Creating archive directory... 2024-03-25T17:44:49.990 DEBUG:teuthology.orchestra.run.smithi083:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T17:44:50.014 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T17:44:50.044 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-25T17:44:50.050 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-25T17:44:50.050 DEBUG:teuthology.orchestra.run.smithi083:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T17:44:50.081 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T17:44:50.123 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T17:44:50.137 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T17:44:50.260 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T17:44:50.273 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T17:44:50.275 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-25T17:44:50.282 INFO:teuthology.task.internal:Configuring sudo... 2024-03-25T17:44:50.282 DEBUG:teuthology.orchestra.run.smithi083:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T17:44:50.285 DEBUG:teuthology.orchestra.run.smithi103:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T17:44:50.313 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-25T17:44:50.328 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-25T17:44:50.328 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T17:44:50.357 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T17:44:50.371 DEBUG:teuthology.orchestra.run.smithi083:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T17:44:50.454 DEBUG:teuthology.orchestra.run.smithi083:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T17:44:50.513 DEBUG:teuthology.orchestra.run.smithi083:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T17:44:50.597 DEBUG:teuthology.orchestra.run.smithi083:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T17:44:50.640 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-25T17:44:50.640 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T17:44:50.707 DEBUG:teuthology.orchestra.run.smithi103:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T17:44:50.748 DEBUG:teuthology.orchestra.run.smithi103:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T17:44:50.797 DEBUG:teuthology.orchestra.run.smithi103:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T17:44:50.880 DEBUG:teuthology.orchestra.run.smithi103:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T17:44:50.921 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-03-25T17:44:50.921 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T17:44:50.987 DEBUG:teuthology.orchestra.run.smithi083:> sudo service rsyslog restart 2024-03-25T17:44:50.989 DEBUG:teuthology.orchestra.run.smithi103:> sudo service rsyslog restart 2024-03-25T17:44:51.039 INFO:teuthology.orchestra.run.smithi083.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T17:44:51.079 INFO:teuthology.orchestra.run.smithi103.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T17:44:51.544 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-25T17:44:51.552 INFO:teuthology.task.internal:Starting timer... 2024-03-25T17:44:51.552 INFO:teuthology.run_tasks:Running task pcp... 2024-03-25T17:44:51.567 INFO:teuthology.run_tasks:Running task selinux... 2024-03-25T17:44:51.575 DEBUG:teuthology.orchestra.run.smithi083:> sudo service auditd rotate 2024-03-25T17:44:51.632 INFO:teuthology.orchestra.run.smithi083.stdout:Rotating logs: 2024-03-25T17:44:51.633 DEBUG:teuthology.orchestra.run.smithi103:> sudo service auditd rotate 2024-03-25T17:44:51.705 INFO:teuthology.orchestra.run.smithi103.stdout:Rotating logs: 2024-03-25T17:44:51.707 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-25T17:44:51.707 DEBUG:teuthology.orchestra.run.smithi083:> /usr/sbin/getenforce 2024-03-25T17:44:51.731 INFO:teuthology.orchestra.run.smithi083.stdout:Permissive 2024-03-25T17:44:51.731 DEBUG:teuthology.orchestra.run.smithi103:> /usr/sbin/getenforce 2024-03-25T17:44:51.761 INFO:teuthology.orchestra.run.smithi103.stdout:Permissive 2024-03-25T17:44:51.761 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi083.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi103.front.sepia.ceph.com': 'permissive'} 2024-03-25T17:44:51.762 DEBUG:teuthology.orchestra.run.smithi083:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T17:44:51.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:44:51.798 DEBUG:teuthology.orchestra.run.smithi103:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T17:44:51.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:44:51.829 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-25T17:44:51.829 DEBUG:teuthology.orchestra.run.smithi083:> sudo /usr/sbin/setenforce permissive 2024-03-25T17:44:51.874 DEBUG:teuthology.orchestra.run.smithi103:> sudo /usr/sbin/setenforce permissive 2024-03-25T17:44:51.912 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-25T17:44:51.921 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-25T17:44:51.927 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-25T17:44:52.048 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-25T17:44:52.064 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-25T17:44:52.067 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi083.front.sepia.ceph.com,smithi103.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-25T17:49:09.837 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi083.front.sepia.ceph.com'), Remote(name='ubuntu@smithi103.front.sepia.ceph.com')] 2024-03-25T17:49:09.839 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-03-25T17:49:09.840 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:49:09.930 DEBUG:teuthology.orchestra.run.smithi083:> true 2024-03-25T17:49:10.004 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-03-25T17:49:10.005 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-03-25T17:49:10.005 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:49:10.162 DEBUG:teuthology.orchestra.run.smithi103:> true 2024-03-25T17:49:10.236 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-03-25T17:49:10.237 INFO:teuthology.run_tasks:Running task clock... 2024-03-25T17:49:10.248 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-25T17:49:10.248 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T17:49:10.248 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T17:49:10.251 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T17:49:10.251 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T17:49:10.279 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T17:49:10.296 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T17:49:10.308 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T17:49:10.324 INFO:teuthology.orchestra.run.smithi083.stderr:sudo: ntpd: command not found 2024-03-25T17:49:10.325 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T17:49:10.338 INFO:teuthology.orchestra.run.smithi083.stdout:506 Cannot talk to daemon 2024-03-25T17:49:10.353 INFO:teuthology.orchestra.run.smithi103.stderr:sudo: ntpd: command not found 2024-03-25T17:49:10.354 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T17:49:10.367 INFO:teuthology.orchestra.run.smithi103.stdout:506 Cannot talk to daemon 2024-03-25T17:49:10.371 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T17:49:10.383 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T17:49:10.400 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T17:49:10.433 INFO:teuthology.orchestra.run.smithi083.stderr:bash: line 1: ntpq: command not found 2024-03-25T17:49:10.436 INFO:teuthology.orchestra.run.smithi083.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T17:49:10.436 INFO:teuthology.orchestra.run.smithi083.stdout:=============================================================================== 2024-03-25T17:49:10.436 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:49:10.436 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:49:10.437 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:49:10.437 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:49:10.454 INFO:teuthology.orchestra.run.smithi103.stderr:bash: line 1: ntpq: command not found 2024-03-25T17:49:10.458 INFO:teuthology.orchestra.run.smithi103.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T17:49:10.458 INFO:teuthology.orchestra.run.smithi103.stdout:=============================================================================== 2024-03-25T17:49:10.458 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:49:10.458 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:49:10.458 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:49:10.458 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:49:10.459 INFO:teuthology.run_tasks:Running task install... 2024-03-25T17:49:10.467 DEBUG:teuthology.task.install:project ceph 2024-03-25T17:49:10.467 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-25T17:49:10.467 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-25T17:49:10.467 INFO:teuthology.task.install:Using flavor: default 2024-03-25T17:49:10.472 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-25T17:49:10.472 INFO:teuthology.task.install:extra packages: [] 2024-03-25T17:49:10.472 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T17:49:10.473 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T17:49:10.475 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T17:49:10.475 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T17:49:10.696 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T17:49:10.697 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T17:49:10.700 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T17:49:10.700 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T17:49:11.090 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T17:49:11.090 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-03-25T17:49:11.090 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T17:49:11.092 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T17:49:11.092 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-25T17:49:11.093 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T17:49:11.122 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T17:49:11.122 DEBUG:teuthology.orchestra.run.smithi103:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T17:49:11.123 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T17:49:11.123 DEBUG:teuthology.orchestra.run.smithi083:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T17:49:11.191 DEBUG:teuthology.orchestra.run.smithi103:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T17:49:11.192 DEBUG:teuthology.orchestra.run.smithi083:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T17:49:11.271 DEBUG:teuthology.orchestra.run.smithi103:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T17:49:11.272 DEBUG:teuthology.orchestra.run.smithi083:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T17:49:11.338 INFO:teuthology.orchestra.run.smithi103.stdout:check_obsoletes = 1 2024-03-25T17:49:11.340 INFO:teuthology.orchestra.run.smithi083.stdout:check_obsoletes = 1 2024-03-25T17:49:11.340 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum clean all 2024-03-25T17:49:11.341 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum clean all 2024-03-25T17:49:11.590 INFO:teuthology.orchestra.run.smithi083.stdout:74 files removed 2024-03-25T17:49:11.606 INFO:teuthology.orchestra.run.smithi103.stdout:74 files removed 2024-03-25T17:49:11.618 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-radosgw 2024-03-25T17:49:11.632 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-radosgw 2024-03-25T17:49:12.912 INFO:teuthology.orchestra.run.smithi083.stdout:ceph packages for x86_64 82 kB/s | 82 kB 00:00 2024-03-25T17:49:13.169 INFO:teuthology.orchestra.run.smithi103.stdout:ceph packages for x86_64 66 kB/s | 82 kB 00:01 2024-03-25T17:49:13.787 INFO:teuthology.orchestra.run.smithi083.stdout:ceph noarch packages 12 kB/s | 10 kB 00:00 2024-03-25T17:49:14.128 INFO:teuthology.orchestra.run.smithi103.stdout:ceph noarch packages 11 kB/s | 10 kB 00:00 2024-03-25T17:49:14.671 INFO:teuthology.orchestra.run.smithi083.stdout:ceph source packages 2.2 kB/s | 1.9 kB 00:00 2024-03-25T17:49:15.021 INFO:teuthology.orchestra.run.smithi103.stdout:ceph source packages 2.2 kB/s | 1.9 kB 00:00 2024-03-25T17:49:15.090 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-25T17:49:15.439 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-25T17:49:16.499 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS-9 - BaseOS 7.4 MB/s | 2.0 MB 00:00 2024-03-25T17:49:16.940 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS-9 - BaseOS 5.1 MB/s | 2.0 MB 00:00 2024-03-25T17:49:24.864 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - BaseOS 1.0 MB/s | 8.0 MB 00:08 2024-03-25T17:49:25.546 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - BaseOS 995 kB/s | 8.0 MB 00:08 2024-03-25T17:49:33.561 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - AppStream 2.5 MB/s | 19 MB 00:07 2024-03-25T17:49:34.693 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - AppStream 2.3 MB/s | 19 MB 00:08 2024-03-25T17:49:44.424 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - CRB 850 kB/s | 6.1 MB 00:07 2024-03-25T17:49:45.197 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - CRB 889 kB/s | 6.1 MB 00:07 2024-03-25T17:49:46.669 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 16 kB 00:01 2024-03-25T17:49:48.013 INFO:teuthology.orchestra.run.smithi083.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-03-25T17:49:53.196 INFO:teuthology.orchestra.run.smithi083.stdout:lab-extras 4.7 kB/s | 1.7 kB 00:00 2024-03-25T17:49:55.638 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T17:49:55.649 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T17:49:55.649 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-03-25T17:49:55.649 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T17:49:55.649 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T17:49:55.649 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T17:49:55.649 INFO:teuthology.orchestra.run.smithi083.stdout:Upgrading: 2024-03-25T17:49:55.649 INFO:teuthology.orchestra.run.smithi083.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T17:49:55.649 INFO:teuthology.orchestra.run.smithi083.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T17:49:55.649 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-03-25T17:49:55.649 INFO:teuthology.orchestra.run.smithi083.stdout: boost-program-options 2024-03-25T17:49:55.650 INFO:teuthology.orchestra.run.smithi083.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T17:49:55.650 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T17:49:55.650 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:49:55.650 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T17:49:55.650 INFO:teuthology.orchestra.run.smithi083.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T17:49:55.650 INFO:teuthology.orchestra.run.smithi083.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T17:49:55.650 INFO:teuthology.orchestra.run.smithi083.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T17:49:55.650 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T17:49:55.650 INFO:teuthology.orchestra.run.smithi083.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T17:49:55.650 INFO:teuthology.orchestra.run.smithi083.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T17:49:55.650 INFO:teuthology.orchestra.run.smithi083.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T17:49:55.650 INFO:teuthology.orchestra.run.smithi083.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T17:49:55.651 INFO:teuthology.orchestra.run.smithi083.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T17:49:55.651 INFO:teuthology.orchestra.run.smithi083.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T17:49:55.651 INFO:teuthology.orchestra.run.smithi083.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T17:49:55.651 INFO:teuthology.orchestra.run.smithi083.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T17:49:55.651 INFO:teuthology.orchestra.run.smithi083.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T17:49:55.651 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-argparse 2024-03-25T17:49:55.651 INFO:teuthology.orchestra.run.smithi083.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T17:49:55.651 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T17:49:55.651 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T17:49:55.651 INFO:teuthology.orchestra.run.smithi083.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T17:49:55.651 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T17:49:55.651 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T17:49:55.651 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T17:49:55.652 INFO:teuthology.orchestra.run.smithi083.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-25T17:49:55.652 INFO:teuthology.orchestra.run.smithi083.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-25T17:49:55.652 INFO:teuthology.orchestra.run.smithi083.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T17:49:55.652 INFO:teuthology.orchestra.run.smithi083.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T17:49:55.652 INFO:teuthology.orchestra.run.smithi083.stdout:Installing weak dependencies: 2024-03-25T17:49:55.652 INFO:teuthology.orchestra.run.smithi083.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-25T17:49:55.652 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:49:55.652 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T17:49:55.652 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T17:49:55.652 INFO:teuthology.orchestra.run.smithi083.stdout:Install 30 Packages 2024-03-25T17:49:55.652 INFO:teuthology.orchestra.run.smithi083.stdout:Upgrade 2 Packages 2024-03-25T17:49:55.652 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:49:55.653 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 58 M 2024-03-25T17:49:55.653 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T17:50:05.079 INFO:teuthology.orchestra.run.smithi083.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 601 kB/s | 5.3 MB 00:09 2024-03-25T17:50:05.179 INFO:teuthology.orchestra.run.smithi083.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 252 kB/s | 25 kB 00:00 2024-03-25T17:50:06.538 INFO:teuthology.orchestra.run.smithi083.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 546 kB/s | 742 kB 00:01 2024-03-25T17:50:07.979 INFO:teuthology.orchestra.run.smithi083.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 346 kB/s | 498 kB 00:01 2024-03-25T17:50:17.663 INFO:teuthology.orchestra.run.smithi083.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 539 kB/s | 5.1 MB 00:09 2024-03-25T17:50:17.763 INFO:teuthology.orchestra.run.smithi083.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 454 kB/s | 45 kB 00:00 2024-03-25T17:50:18.021 INFO:teuthology.orchestra.run.smithi083.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 508 kB/s | 131 kB 00:00 2024-03-25T17:50:18.388 INFO:teuthology.orchestra.run.smithi083.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 441 kB/s | 162 kB 00:00 2024-03-25T17:50:19.031 INFO:teuthology.orchestra.run.smithi083.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 499 kB/s | 320 kB 00:00 2024-03-25T17:50:19.531 INFO:teuthology.orchestra.run.smithi083.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 596 kB/s | 298 kB 00:00 2024-03-25T17:50:19.723 INFO:teuthology.orchestra.run.smithi083.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 519 kB/s | 99 kB 00:00 2024-03-25T17:50:19.823 INFO:teuthology.orchestra.run.smithi083.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 1.0 MB/s | 106 kB 00:00 2024-03-25T17:50:19.940 INFO:teuthology.orchestra.run.smithi083.stdout:(13/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 416 kB/s | 9.7 MB 00:23 2024-03-25T17:50:19.965 INFO:teuthology.orchestra.run.smithi083.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.1 MB/s | 159 kB 00:00 2024-03-25T17:50:19.982 INFO:teuthology.orchestra.run.smithi083.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-25T17:50:20.032 INFO:teuthology.orchestra.run.smithi083.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-03-25T17:50:20.057 INFO:teuthology.orchestra.run.smithi083.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-03-25T17:50:20.083 INFO:teuthology.orchestra.run.smithi083.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 830 kB/s | 41 kB 00:00 2024-03-25T17:50:20.117 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:50:20.117 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:50:20.161 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:50:20.162 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:50:20.261 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:50:20.273 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:50:20.326 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:50:20.339 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:50:20.456 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:50:20.481 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:50:20.943 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:50:20.962 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:50:21.272 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:50:21.288 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:50:21.312 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:50:21.333 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:50:21.403 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:50:21.428 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:50:21.549 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:50:21.567 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:50:21.567 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:50:21.585 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:50:21.633 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:50:21.649 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:50:21.754 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:50:21.777 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:50:21.918 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:50:21.937 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:50:22.264 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:50:22.290 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:50:22.341 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:50:22.366 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:50:22.478 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:50:22.513 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:50:22.547 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:50:22.566 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:50:22.566 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:50:22.584 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:50:22.585 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:50:22.618 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:50:22.661 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:50:22.669 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:50:22.685 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:50:22.692 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:50:22.771 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:50:22.779 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:50:22.910 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:50:22.933 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:50:22.999 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:50:23.010 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:50:23.033 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:50:23.043 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:50:23.117 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:50:23.152 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:50:23.258 INFO:teuthology.orchestra.run.smithi083.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 69 kB/s | 221 kB 00:03 2024-03-25T17:50:23.294 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:50:23.335 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:50:23.400 INFO:teuthology.orchestra.run.smithi083.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 11 kB/s | 36 kB 00:03 2024-03-25T17:50:23.422 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:50:23.488 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:50:23.616 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:50:24.025 INFO:teuthology.orchestra.run.smithi083.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 495 kB/s | 309 kB 00:00 2024-03-25T17:50:24.342 INFO:teuthology.orchestra.run.smithi083.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 15 MB/s | 4.4 MB 00:00 2024-03-25T17:50:24.343 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:50:24.409 INFO:teuthology.orchestra.run.smithi083.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 390 kB/s | 26 kB 00:00 2024-03-25T17:50:24.476 INFO:teuthology.orchestra.run.smithi083.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 728 kB/s | 49 kB 00:00 2024-03-25T17:50:24.534 INFO:teuthology.orchestra.run.smithi083.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-25T17:50:24.618 INFO:teuthology.orchestra.run.smithi083.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 9.9 MB/s | 838 kB 00:00 2024-03-25T17:50:24.694 INFO:teuthology.orchestra.run.smithi083.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-03-25T17:50:24.700 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:50:24.744 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:50:24.827 INFO:teuthology.orchestra.run.smithi083.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-03-25T17:50:24.850 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:50:24.994 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:50:25.013 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:50:25.077 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:50:25.200 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:50:25.363 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:50:25.715 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:50:25.790 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:50:25.930 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:50:25.986 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:50:26.003 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:50:26.021 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:50:26.076 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:50:26.097 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:50:26.169 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:50:26.311 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:50:26.387 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:50:26.435 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:50:26.514 INFO:teuthology.orchestra.run.smithi083.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:50:26.602 INFO:teuthology.orchestra.run.smithi083.stdout:(29/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 14 kB/s | 47 kB 00:03 2024-03-25T17:50:30.877 INFO:teuthology.orchestra.run.smithi083.stdout:(30/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 749 kB/s | 3.1 MB 00:04 2024-03-25T17:50:36.310 INFO:teuthology.orchestra.run.smithi083.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 301 kB/s | 3.4 MB 00:11 2024-03-25T17:51:11.365 INFO:teuthology.orchestra.run.smithi083.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 270 kB/s | 20 MB 01:15 2024-03-25T17:51:11.373 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:51:11.373 INFO:teuthology.orchestra.run.smithi083.stdout:Total 782 kB/s | 58 MB 01:15 2024-03-25T17:51:11.404 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T17:51:11.442 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T17:51:11.442 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T17:51:11.915 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T17:51:11.915 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T17:51:13.433 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T17:51:13.519 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T17:51:13.552 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T17:51:13.578 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T17:51:13.601 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T17:51:13.617 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T17:51:13.810 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T17:51:13.864 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T17:51:14.184 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T17:51:14.272 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T17:51:14.363 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T17:51:14.452 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T17:51:14.482 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T17:51:14.546 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T17:51:14.582 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T17:51:14.626 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T17:51:14.693 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T17:51:14.718 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T17:51:14.765 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T17:51:14.803 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-25T17:51:14.958 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T17:51:14.995 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T17:51:15.041 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T17:51:15.075 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T17:51:15.123 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T17:51:15.163 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T17:51:15.529 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T17:51:15.557 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T17:51:15.646 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T17:51:15.679 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T17:51:15.717 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T17:51:16.166 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T17:51:16.201 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T17:51:16.272 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T17:51:16.340 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T17:51:16.370 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T17:51:18.515 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T17:51:18.684 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T17:51:19.234 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T17:51:19.278 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T17:51:19.368 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T17:51:19.368 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T17:51:19.368 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T17:51:19.369 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:51:19.405 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T17:51:28.770 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T17:51:28.771 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /sys 2024-03-25T17:51:28.771 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /proc 2024-03-25T17:51:28.771 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /mnt 2024-03-25T17:51:28.771 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /var/tmp 2024-03-25T17:51:28.771 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /home 2024-03-25T17:51:28.771 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /root 2024-03-25T17:51:28.771 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /tmp 2024-03-25T17:51:28.771 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:51:29.699 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T17:51:29.723 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T17:51:29.723 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:51:29.723 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T17:51:29.723 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T17:51:29.723 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T17:51:29.723 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:51:29.725 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T17:51:29.813 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T17:51:29.813 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T17:51:30.977 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T17:51:30.977 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T17:51:30.977 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T17:51:30.977 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T17:51:30.978 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T17:51:30.978 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T17:51:30.978 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T17:51:30.978 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T17:51:30.978 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T17:51:30.978 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T17:51:30.978 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T17:51:30.978 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T17:51:30.978 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T17:51:30.978 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T17:51:30.978 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T17:51:30.978 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T17:51:30.978 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T17:51:30.979 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T17:51:30.979 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T17:51:30.979 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T17:51:30.979 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-25T17:51:30.979 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-25T17:51:30.979 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-25T17:51:30.979 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T17:51:30.979 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T17:51:30.979 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T17:51:30.979 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T17:51:30.979 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T17:51:30.980 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T17:51:30.980 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T17:51:30.980 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T17:51:30.980 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T17:51:30.981 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T17:51:30.981 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T17:51:32.094 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T17:51:32.094 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:51:32.094 INFO:teuthology.orchestra.run.smithi083.stdout:Upgraded: 2024-03-25T17:51:32.094 INFO:teuthology.orchestra.run.smithi083.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:51:32.094 INFO:teuthology.orchestra.run.smithi083.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:51:32.095 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T17:51:32.095 INFO:teuthology.orchestra.run.smithi083.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T17:51:32.095 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:51:32.095 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:51:32.095 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:51:32.095 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:51:32.095 INFO:teuthology.orchestra.run.smithi083.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T17:51:32.095 INFO:teuthology.orchestra.run.smithi083.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T17:51:32.095 INFO:teuthology.orchestra.run.smithi083.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T17:51:32.095 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:51:32.095 INFO:teuthology.orchestra.run.smithi083.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T17:51:32.095 INFO:teuthology.orchestra.run.smithi083.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T17:51:32.095 INFO:teuthology.orchestra.run.smithi083.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T17:51:32.095 INFO:teuthology.orchestra.run.smithi083.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:51:32.096 INFO:teuthology.orchestra.run.smithi083.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T17:51:32.096 INFO:teuthology.orchestra.run.smithi083.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:51:32.096 INFO:teuthology.orchestra.run.smithi083.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T17:51:32.096 INFO:teuthology.orchestra.run.smithi083.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T17:51:32.096 INFO:teuthology.orchestra.run.smithi083.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T17:51:32.096 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:51:32.096 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:51:32.096 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:51:32.096 INFO:teuthology.orchestra.run.smithi083.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T17:51:32.096 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:51:32.096 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:51:32.096 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:51:32.096 INFO:teuthology.orchestra.run.smithi083.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-25T17:51:32.097 INFO:teuthology.orchestra.run.smithi083.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-25T17:51:32.097 INFO:teuthology.orchestra.run.smithi083.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-25T17:51:32.097 INFO:teuthology.orchestra.run.smithi083.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T17:51:32.097 INFO:teuthology.orchestra.run.smithi083.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T17:51:32.097 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:51:32.097 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T17:51:32.372 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-test 2024-03-25T17:51:32.924 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:40 ago on Mon 25 Mar 2024 05:49:52 PM UTC. 2024-03-25T17:51:33.076 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T17:51:33.077 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T17:51:33.077 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-03-25T17:51:33.077 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T17:51:33.077 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T17:51:33.077 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T17:51:33.077 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-03-25T17:51:33.077 INFO:teuthology.orchestra.run.smithi083.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T17:51:33.078 INFO:teuthology.orchestra.run.smithi083.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T17:51:33.078 INFO:teuthology.orchestra.run.smithi083.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T17:51:33.078 INFO:teuthology.orchestra.run.smithi083.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T17:51:33.078 INFO:teuthology.orchestra.run.smithi083.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T17:51:33.078 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:51:33.078 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T17:51:33.078 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T17:51:33.078 INFO:teuthology.orchestra.run.smithi083.stdout:Install 6 Packages 2024-03-25T17:51:33.078 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:51:33.079 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 48 M 2024-03-25T17:51:33.079 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 204 M 2024-03-25T17:51:33.079 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T17:51:33.281 INFO:teuthology.orchestra.run.smithi083.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.5 MB/s | 217 kB 00:00 2024-03-25T17:51:33.381 INFO:teuthology.orchestra.run.smithi083.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 2.9 MB/s | 300 kB 00:00 2024-03-25T17:51:33.439 INFO:teuthology.orchestra.run.smithi083.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 63 kB 00:00 2024-03-25T17:51:33.615 INFO:teuthology.orchestra.run.smithi083.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2024-03-25T17:51:35.340 INFO:teuthology.orchestra.run.smithi083.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 74 kB/s | 163 kB 00:02 2024-03-25T17:53:47.125 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - Extras packages 66 B/s | 16 kB 04:00 2024-03-25T17:53:49.466 INFO:teuthology.orchestra.run.smithi103.stdout:Extra Packages for Enterprise Linux 9.3 MB/s | 21 MB 00:02 2024-03-25T17:53:53.733 INFO:teuthology.orchestra.run.smithi083.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 344 kB/s | 47 MB 02:20 2024-03-25T17:53:53.735 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:53:53.735 INFO:teuthology.orchestra.run.smithi083.stdout:Total 351 kB/s | 48 MB 02:20 2024-03-25T17:53:53.746 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T17:53:53.766 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T17:53:53.766 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T17:53:54.144 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T17:53:54.145 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T17:53:54.641 INFO:teuthology.orchestra.run.smithi103.stdout:lab-extras 4.9 kB/s | 1.7 kB 00:00 2024-03-25T17:53:54.738 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T17:53:54.799 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T17:53:54.840 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T17:53:54.880 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T17:53:54.930 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T17:53:54.961 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T17:53:57.088 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-25T17:53:57.101 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T17:53:57.101 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-03-25T17:53:57.101 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T17:53:57.101 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-25T17:53:57.101 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T17:53:57.102 INFO:teuthology.orchestra.run.smithi103.stdout:Upgrading: 2024-03-25T17:53:57.102 INFO:teuthology.orchestra.run.smithi103.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T17:53:57.102 INFO:teuthology.orchestra.run.smithi103.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T17:53:57.102 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-03-25T17:53:57.102 INFO:teuthology.orchestra.run.smithi103.stdout: boost-program-options 2024-03-25T17:53:57.102 INFO:teuthology.orchestra.run.smithi103.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T17:53:57.102 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T17:53:57.102 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:53:57.102 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T17:53:57.102 INFO:teuthology.orchestra.run.smithi103.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T17:53:57.102 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T17:53:57.102 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T17:53:57.103 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T17:53:57.103 INFO:teuthology.orchestra.run.smithi103.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T17:53:57.103 INFO:teuthology.orchestra.run.smithi103.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T17:53:57.103 INFO:teuthology.orchestra.run.smithi103.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T17:53:57.103 INFO:teuthology.orchestra.run.smithi103.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T17:53:57.103 INFO:teuthology.orchestra.run.smithi103.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T17:53:57.103 INFO:teuthology.orchestra.run.smithi103.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T17:53:57.103 INFO:teuthology.orchestra.run.smithi103.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T17:53:57.103 INFO:teuthology.orchestra.run.smithi103.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T17:53:57.103 INFO:teuthology.orchestra.run.smithi103.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T17:53:57.103 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-argparse 2024-03-25T17:53:57.103 INFO:teuthology.orchestra.run.smithi103.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T17:53:57.103 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T17:53:57.103 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T17:53:57.104 INFO:teuthology.orchestra.run.smithi103.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T17:53:57.104 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T17:53:57.104 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T17:53:57.104 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T17:53:57.104 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-25T17:53:57.104 INFO:teuthology.orchestra.run.smithi103.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-25T17:53:57.104 INFO:teuthology.orchestra.run.smithi103.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T17:53:57.104 INFO:teuthology.orchestra.run.smithi103.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T17:53:57.104 INFO:teuthology.orchestra.run.smithi103.stdout:Installing weak dependencies: 2024-03-25T17:53:57.104 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-25T17:53:57.104 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T17:53:57.104 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-25T17:53:57.104 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T17:53:57.104 INFO:teuthology.orchestra.run.smithi103.stdout:Install 30 Packages 2024-03-25T17:53:57.104 INFO:teuthology.orchestra.run.smithi103.stdout:Upgrade 2 Packages 2024-03-25T17:53:57.105 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T17:53:57.105 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 58 M 2024-03-25T17:53:57.105 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-25T17:53:59.422 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T17:53:59.502 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T17:54:01.912 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T17:54:01.912 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T17:54:01.912 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T17:54:01.912 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T17:54:01.912 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T17:54:01.912 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T17:54:02.314 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T17:54:02.314 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:54:02.314 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T17:54:02.314 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:54:02.314 INFO:teuthology.orchestra.run.smithi083.stdout: jq-1.6-16.el9.x86_64 2024-03-25T17:54:02.314 INFO:teuthology.orchestra.run.smithi083.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:54:02.314 INFO:teuthology.orchestra.run.smithi083.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T17:54:02.314 INFO:teuthology.orchestra.run.smithi083.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T17:54:02.315 INFO:teuthology.orchestra.run.smithi083.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T17:54:02.315 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:54:02.315 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T17:54:02.634 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph 2024-03-25T17:54:03.201 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:04:11 ago on Mon 25 Mar 2024 05:49:52 PM UTC. 2024-03-25T17:54:03.359 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T17:54:03.362 INFO:teuthology.orchestra.run.smithi083.stdout:========================================================================================= 2024-03-25T17:54:03.362 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-03-25T17:54:03.362 INFO:teuthology.orchestra.run.smithi083.stdout:========================================================================================= 2024-03-25T17:54:03.362 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T17:54:03.362 INFO:teuthology.orchestra.run.smithi083.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T17:54:03.362 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-03-25T17:54:03.362 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T17:54:03.362 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T17:54:03.362 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T17:54:03.362 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T17:54:03.363 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T17:54:03.363 INFO:teuthology.orchestra.run.smithi083.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T17:54:03.363 INFO:teuthology.orchestra.run.smithi083.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T17:54:03.363 INFO:teuthology.orchestra.run.smithi083.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T17:54:03.363 INFO:teuthology.orchestra.run.smithi083.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T17:54:03.363 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T17:54:03.363 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T17:54:03.363 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T17:54:03.363 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T17:54:03.363 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T17:54:03.363 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T17:54:03.363 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T17:54:03.363 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T17:54:03.364 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T17:54:03.364 INFO:teuthology.orchestra.run.smithi083.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T17:54:03.364 INFO:teuthology.orchestra.run.smithi083.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T17:54:03.364 INFO:teuthology.orchestra.run.smithi083.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T17:54:03.364 INFO:teuthology.orchestra.run.smithi083.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T17:54:03.364 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T17:54:03.364 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T17:54:03.364 INFO:teuthology.orchestra.run.smithi083.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T17:54:03.364 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T17:54:03.364 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T17:54:03.364 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T17:54:03.364 INFO:teuthology.orchestra.run.smithi083.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T17:54:03.364 INFO:teuthology.orchestra.run.smithi083.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T17:54:03.364 INFO:teuthology.orchestra.run.smithi083.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T17:54:03.364 INFO:teuthology.orchestra.run.smithi083.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T17:54:03.364 INFO:teuthology.orchestra.run.smithi083.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T17:54:03.365 INFO:teuthology.orchestra.run.smithi083.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T17:54:03.365 INFO:teuthology.orchestra.run.smithi083.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T17:54:03.365 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:54:03.365 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T17:54:03.365 INFO:teuthology.orchestra.run.smithi083.stdout:========================================================================================= 2024-03-25T17:54:03.365 INFO:teuthology.orchestra.run.smithi083.stdout:Install 36 Packages 2024-03-25T17:54:03.365 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:54:03.365 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 30 M 2024-03-25T17:54:03.366 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 107 M 2024-03-25T17:54:03.366 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T17:54:04.253 INFO:teuthology.orchestra.run.smithi083.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 14 kB/s | 6.6 kB 00:00 2024-03-25T17:54:09.154 INFO:teuthology.orchestra.run.smithi083.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 283 kB/s | 1.5 MB 00:05 2024-03-25T17:54:09.662 INFO:teuthology.orchestra.run.smithi083.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 369 kB/s | 2.1 MB 00:05 2024-03-25T17:54:09.899 INFO:teuthology.orchestra.run.smithi103.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 443 kB/s | 5.3 MB 00:12 2024-03-25T17:54:10.033 INFO:teuthology.orchestra.run.smithi103.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 189 kB/s | 25 kB 00:00 2024-03-25T17:54:10.562 INFO:teuthology.orchestra.run.smithi083.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 275 kB/s | 247 kB 00:00 2024-03-25T17:54:10.704 INFO:teuthology.orchestra.run.smithi083.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.3 MB/s | 187 kB 00:00 2024-03-25T17:54:10.746 INFO:teuthology.orchestra.run.smithi083.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T17:54:10.780 INFO:teuthology.orchestra.run.smithi083.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 945 kB/s | 32 kB 00:00 2024-03-25T17:54:10.813 INFO:teuthology.orchestra.run.smithi083.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-25T17:54:10.847 INFO:teuthology.orchestra.run.smithi083.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-25T17:54:10.889 INFO:teuthology.orchestra.run.smithi083.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-25T17:54:10.956 INFO:teuthology.orchestra.run.smithi083.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-25T17:54:10.998 INFO:teuthology.orchestra.run.smithi083.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2024-03-25T17:54:11.040 INFO:teuthology.orchestra.run.smithi083.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-03-25T17:54:11.081 INFO:teuthology.orchestra.run.smithi083.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-25T17:54:11.174 INFO:teuthology.orchestra.run.smithi083.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 1.3 MB/s | 125 kB 00:00 2024-03-25T17:54:11.508 INFO:teuthology.orchestra.run.smithi103.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 503 kB/s | 742 kB 00:01 2024-03-25T17:54:11.774 INFO:teuthology.orchestra.run.smithi083.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 37 kB/s | 22 kB 00:00 2024-03-25T17:54:12.091 INFO:teuthology.orchestra.run.smithi083.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 477 kB/s | 151 kB 00:00 2024-03-25T17:54:12.241 INFO:teuthology.orchestra.run.smithi083.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 290 kB/s | 43 kB 00:00 2024-03-25T17:54:12.333 INFO:teuthology.orchestra.run.smithi103.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 604 kB/s | 498 kB 00:00 2024-03-25T17:54:12.399 INFO:teuthology.orchestra.run.smithi083.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-03-25T17:54:12.558 INFO:teuthology.orchestra.run.smithi083.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.2 MB/s | 358 kB 00:00 2024-03-25T17:54:12.634 INFO:teuthology.orchestra.run.smithi083.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 142 kB/s | 11 kB 00:00 2024-03-25T17:54:12.709 INFO:teuthology.orchestra.run.smithi083.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 235 kB/s | 18 kB 00:00 2024-03-25T17:54:12.809 INFO:teuthology.orchestra.run.smithi083.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 232 kB/s | 23 kB 00:00 2024-03-25T17:54:12.893 INFO:teuthology.orchestra.run.smithi083.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 233 kB/s | 19 kB 00:00 2024-03-25T17:54:12.985 INFO:teuthology.orchestra.run.smithi083.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 214 kB/s | 20 kB 00:00 2024-03-25T17:54:13.102 INFO:teuthology.orchestra.run.smithi083.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 397 kB/s | 46 kB 00:00 2024-03-25T17:54:13.211 INFO:teuthology.orchestra.run.smithi083.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 727 kB/s | 79 kB 00:00 2024-03-25T17:54:13.344 INFO:teuthology.orchestra.run.smithi083.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2024-03-25T17:54:13.436 INFO:teuthology.orchestra.run.smithi083.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 179 kB/s | 16 kB 00:00 2024-03-25T17:54:13.537 INFO:teuthology.orchestra.run.smithi083.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 899 kB/s | 90 kB 00:00 2024-03-25T17:54:13.670 INFO:teuthology.orchestra.run.smithi083.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 268 kB/s | 36 kB 00:00 2024-03-25T17:54:13.779 INFO:teuthology.orchestra.run.smithi083.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-03-25T17:54:13.921 INFO:teuthology.orchestra.run.smithi083.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.9 MB/s | 427 kB 00:00 2024-03-25T17:54:14.021 INFO:teuthology.orchestra.run.smithi083.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 199 kB/s | 20 kB 00:00 2024-03-25T17:54:16.063 INFO:teuthology.orchestra.run.smithi083.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 406 kB/s | 4.7 MB 00:11 2024-03-25T17:54:19.442 INFO:teuthology.orchestra.run.smithi103.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 455 kB/s | 9.7 MB 00:21 2024-03-25T17:54:19.616 INFO:teuthology.orchestra.run.smithi103.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 258 kB/s | 45 kB 00:00 2024-03-25T17:54:20.350 INFO:teuthology.orchestra.run.smithi103.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 179 kB/s | 131 kB 00:00 2024-03-25T17:54:21.282 INFO:teuthology.orchestra.run.smithi103.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 174 kB/s | 162 kB 00:00 2024-03-25T17:54:23.717 INFO:teuthology.orchestra.run.smithi103.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 132 kB/s | 320 kB 00:02 2024-03-25T17:54:25.100 INFO:teuthology.orchestra.run.smithi103.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 215 kB/s | 298 kB 00:01 2024-03-25T17:54:25.176 INFO:teuthology.orchestra.run.smithi103.stdout:(11/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 407 kB/s | 5.1 MB 00:12 2024-03-25T17:54:25.304 INFO:teuthology.orchestra.run.smithi103.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 830 kB/s | 106 kB 00:00 2024-03-25T17:54:25.351 INFO:teuthology.orchestra.run.smithi103.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.3 MB/s | 159 kB 00:00 2024-03-25T17:54:25.385 INFO:teuthology.orchestra.run.smithi103.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-25T17:54:25.443 INFO:teuthology.orchestra.run.smithi103.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-25T17:54:25.485 INFO:teuthology.orchestra.run.smithi103.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-25T17:54:25.519 INFO:teuthology.orchestra.run.smithi103.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-25T17:54:25.679 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:54:26.033 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:54:26.099 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:54:26.144 INFO:teuthology.orchestra.run.smithi103.stdout:(18/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 95 kB/s | 99 kB 00:01 2024-03-25T17:54:26.236 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:54:26.249 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:54:26.330 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:54:26.363 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:54:26.395 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:54:26.742 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:54:26.895 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:54:27.076 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:54:27.094 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:54:27.163 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:54:27.199 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:54:27.220 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:54:27.239 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:54:27.309 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:54:27.326 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:54:27.340 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:54:27.361 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:54:27.428 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:54:27.431 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:54:27.449 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:54:27.486 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:54:27.578 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:54:27.614 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:54:27.622 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:54:27.667 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:54:27.734 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:54:27.753 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:54:27.809 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:54:27.828 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:54:27.949 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:54:27.970 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:54:27.971 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:54:28.016 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:54:28.051 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:54:28.085 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:54:28.096 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:54:28.141 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:54:28.187 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:54:28.245 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:54:28.251 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:54:28.287 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:54:28.298 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:54:28.316 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:54:28.445 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:54:28.454 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:54:28.532 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:54:28.545 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:54:28.573 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:54:28.576 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:54:28.664 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:54:28.743 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:54:28.819 INFO:teuthology.orchestra.run.smithi103.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 67 kB/s | 221 kB 00:03 2024-03-25T17:54:28.844 INFO:teuthology.orchestra.run.smithi103.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 13 kB/s | 36 kB 00:02 2024-03-25T17:54:28.998 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:54:29.269 INFO:teuthology.orchestra.run.smithi103.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 727 kB/s | 309 kB 00:00 2024-03-25T17:54:29.350 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:54:29.414 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:54:29.603 INFO:teuthology.orchestra.run.smithi103.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-03-25T17:54:29.604 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:54:29.678 INFO:teuthology.orchestra.run.smithi103.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 350 kB/s | 26 kB 00:00 2024-03-25T17:54:29.754 INFO:teuthology.orchestra.run.smithi103.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 648 kB/s | 49 kB 00:00 2024-03-25T17:54:29.854 INFO:teuthology.orchestra.run.smithi103.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 675 kB/s | 67 kB 00:00 2024-03-25T17:54:29.971 INFO:teuthology.orchestra.run.smithi103.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 8.8 MB/s | 838 kB 00:00 2024-03-25T17:54:30.046 INFO:teuthology.orchestra.run.smithi103.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-03-25T17:54:30.146 INFO:teuthology.orchestra.run.smithi103.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-03-25T17:54:30.148 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:54:30.475 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:54:30.494 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:54:30.546 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:54:30.580 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:54:30.697 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:54:30.716 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:54:30.782 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:54:30.913 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:54:30.954 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:54:31.041 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:54:31.116 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:54:31.253 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:54:31.273 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:54:31.349 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:54:31.395 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:54:31.489 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:54:31.545 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:54:31.564 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:54:31.703 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:54:31.793 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:54:31.829 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:54:31.906 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:54:32.005 INFO:teuthology.orchestra.run.smithi103.stdout:(29/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 15 kB/s | 47 kB 00:03 2024-03-25T17:54:40.596 INFO:teuthology.orchestra.run.smithi103.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 331 kB/s | 3.4 MB 00:10 2024-03-25T17:54:42.805 INFO:teuthology.orchestra.run.smithi103.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 297 kB/s | 3.1 MB 00:10 2024-03-25T17:54:47.962 INFO:teuthology.orchestra.run.smithi103.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 404 kB/s | 20 MB 00:50 2024-03-25T17:54:47.971 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:54:47.971 INFO:teuthology.orchestra.run.smithi103.stdout:Total 1.1 MB/s | 58 MB 00:50 2024-03-25T17:54:48.004 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-25T17:54:48.044 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-25T17:54:48.044 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-25T17:54:48.512 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-25T17:54:48.512 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-25T17:54:50.012 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-25T17:54:50.112 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T17:54:50.142 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T17:54:50.168 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T17:54:50.190 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T17:54:50.207 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T17:54:50.366 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T17:54:50.421 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T17:54:50.802 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T17:54:50.836 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T17:54:50.927 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T17:54:51.018 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T17:54:51.046 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T17:54:51.111 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T17:54:51.151 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T17:54:51.199 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T17:54:51.279 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T17:54:51.315 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T17:54:51.362 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T17:54:51.397 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-25T17:54:51.507 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T17:54:51.548 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T17:54:51.588 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T17:54:51.621 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T17:54:51.650 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T17:54:51.682 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T17:54:52.004 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T17:54:52.038 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T17:54:52.119 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T17:54:52.150 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T17:54:52.194 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T17:54:52.653 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T17:54:52.682 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T17:54:52.737 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T17:54:52.804 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T17:54:52.834 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T17:54:55.006 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T17:54:55.047 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T17:54:55.607 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T17:54:55.647 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T17:54:55.730 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T17:54:55.730 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T17:54:55.730 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T17:54:55.730 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T17:54:55.782 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T17:54:57.783 INFO:teuthology.orchestra.run.smithi083.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 353 kB/s | 17 MB 00:48 2024-03-25T17:54:57.787 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:54:57.787 INFO:teuthology.orchestra.run.smithi083.stdout:Total 558 kB/s | 30 MB 00:54 2024-03-25T17:54:57.810 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T17:54:57.852 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T17:54:57.853 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T17:54:58.250 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T17:54:58.250 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T17:54:59.575 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T17:54:59.627 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T17:54:59.666 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T17:54:59.696 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T17:54:59.768 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T17:54:59.804 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T17:54:59.837 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T17:54:59.928 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T17:55:00.007 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T17:55:00.048 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T17:55:00.119 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T17:55:00.173 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T17:55:00.212 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T17:55:00.268 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T17:55:00.324 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T17:55:00.516 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T17:55:00.566 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T17:55:00.604 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T17:55:00.639 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T17:55:00.669 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T17:55:00.746 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T17:55:00.788 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T17:55:00.821 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T17:55:00.862 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T17:55:00.896 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T17:55:00.931 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T17:55:01.020 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T17:55:01.054 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T17:55:01.125 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T17:55:01.263 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T17:55:01.381 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T17:55:01.540 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T17:55:01.571 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T17:55:01.595 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T17:55:01.596 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:55:01.596 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T17:55:01.596 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T17:55:01.596 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T17:55:01.596 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:55:03.030 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T17:55:03.056 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T17:55:03.056 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:55:03.056 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T17:55:03.056 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T17:55:03.056 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T17:55:03.056 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:55:03.497 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T17:55:03.520 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T17:55:03.520 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:55:03.520 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T17:55:03.520 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T17:55:03.521 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T17:55:03.521 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:55:03.732 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T17:55:03.754 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T17:55:03.754 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:55:03.754 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T17:55:03.754 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T17:55:03.755 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T17:55:03.755 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:55:03.820 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T17:55:05.131 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T17:55:05.131 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /sys 2024-03-25T17:55:05.131 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /proc 2024-03-25T17:55:05.131 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /mnt 2024-03-25T17:55:05.131 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /var/tmp 2024-03-25T17:55:05.131 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /home 2024-03-25T17:55:05.131 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /root 2024-03-25T17:55:05.131 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /tmp 2024-03-25T17:55:05.131 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T17:55:05.438 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T17:55:05.438 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T17:55:05.438 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T17:55:05.438 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T17:55:05.438 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T17:55:05.438 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T17:55:05.438 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T17:55:05.438 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T17:55:05.439 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T17:55:05.439 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T17:55:05.439 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T17:55:05.439 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T17:55:05.439 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T17:55:05.439 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T17:55:05.439 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T17:55:05.439 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T17:55:05.439 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T17:55:05.439 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T17:55:05.439 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T17:55:05.439 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T17:55:05.439 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T17:55:05.440 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T17:55:05.440 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T17:55:05.440 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T17:55:05.440 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T17:55:05.440 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T17:55:05.440 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T17:55:05.440 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T17:55:05.440 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T17:55:05.441 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T17:55:05.441 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T17:55:05.441 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T17:55:05.441 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T17:55:05.441 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T17:55:05.441 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T17:55:05.441 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T17:55:06.037 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T17:55:06.061 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T17:55:06.061 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:55:06.062 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T17:55:06.062 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T17:55:06.062 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T17:55:06.062 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T17:55:06.063 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T17:55:06.132 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T17:55:06.132 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T17:55:06.373 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T17:55:06.373 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:55:06.373 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T17:55:06.373 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:06.373 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:06.373 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:06.373 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:55:06.373 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:06.373 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:06.374 INFO:teuthology.orchestra.run.smithi083.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T17:55:06.374 INFO:teuthology.orchestra.run.smithi083.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T17:55:06.374 INFO:teuthology.orchestra.run.smithi083.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T17:55:06.374 INFO:teuthology.orchestra.run.smithi083.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T17:55:06.374 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T17:55:06.374 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T17:55:06.374 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T17:55:06.374 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T17:55:06.374 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T17:55:06.374 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T17:55:06.374 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T17:55:06.374 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T17:55:06.374 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T17:55:06.374 INFO:teuthology.orchestra.run.smithi083.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T17:55:06.375 INFO:teuthology.orchestra.run.smithi083.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T17:55:06.375 INFO:teuthology.orchestra.run.smithi083.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T17:55:06.375 INFO:teuthology.orchestra.run.smithi083.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T17:55:06.375 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T17:55:06.375 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T17:55:06.375 INFO:teuthology.orchestra.run.smithi083.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T17:55:06.375 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T17:55:06.375 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T17:55:06.375 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T17:55:06.375 INFO:teuthology.orchestra.run.smithi083.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T17:55:06.375 INFO:teuthology.orchestra.run.smithi083.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T17:55:06.375 INFO:teuthology.orchestra.run.smithi083.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T17:55:06.375 INFO:teuthology.orchestra.run.smithi083.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T17:55:06.375 INFO:teuthology.orchestra.run.smithi083.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T17:55:06.375 INFO:teuthology.orchestra.run.smithi083.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T17:55:06.376 INFO:teuthology.orchestra.run.smithi083.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T17:55:06.376 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:55:06.376 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T17:55:06.738 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-base 2024-03-25T17:55:07.270 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T17:55:07.270 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T17:55:07.270 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T17:55:07.270 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T17:55:07.271 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T17:55:07.271 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T17:55:07.271 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T17:55:07.271 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T17:55:07.271 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T17:55:07.271 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T17:55:07.271 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T17:55:07.271 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T17:55:07.271 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T17:55:07.271 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T17:55:07.271 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T17:55:07.271 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T17:55:07.271 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T17:55:07.272 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T17:55:07.272 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T17:55:07.272 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T17:55:07.272 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-25T17:55:07.272 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-25T17:55:07.272 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-25T17:55:07.272 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T17:55:07.272 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T17:55:07.272 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T17:55:07.273 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T17:55:07.273 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T17:55:07.273 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T17:55:07.273 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T17:55:07.274 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T17:55:07.274 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T17:55:07.274 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T17:55:07.274 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T17:55:07.298 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:05:15 ago on Mon 25 Mar 2024 05:49:52 PM UTC. 2024-03-25T17:55:07.395 INFO:teuthology.orchestra.run.smithi083.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:55:07.448 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T17:55:07.449 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-03-25T17:55:07.449 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T17:55:07.514 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install cephadm 2024-03-25T17:55:08.073 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:05:16 ago on Mon 25 Mar 2024 05:49:52 PM UTC. 2024-03-25T17:55:08.221 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T17:55:08.222 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T17:55:08.222 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-03-25T17:55:08.222 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T17:55:08.222 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T17:55:08.222 INFO:teuthology.orchestra.run.smithi083.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T17:55:08.222 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:55:08.223 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T17:55:08.223 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T17:55:08.223 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T17:55:08.223 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:55:08.223 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 747 k 2024-03-25T17:55:08.223 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 754 k 2024-03-25T17:55:08.223 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T17:55:08.324 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T17:55:08.324 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T17:55:08.324 INFO:teuthology.orchestra.run.smithi103.stdout:Upgraded: 2024-03-25T17:55:08.325 INFO:teuthology.orchestra.run.smithi103.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:08.325 INFO:teuthology.orchestra.run.smithi103.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:08.325 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-25T17:55:08.325 INFO:teuthology.orchestra.run.smithi103.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T17:55:08.325 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:08.325 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:08.325 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:08.325 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:08.325 INFO:teuthology.orchestra.run.smithi103.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T17:55:08.325 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T17:55:08.325 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T17:55:08.325 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:08.325 INFO:teuthology.orchestra.run.smithi103.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T17:55:08.326 INFO:teuthology.orchestra.run.smithi103.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T17:55:08.326 INFO:teuthology.orchestra.run.smithi103.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T17:55:08.326 INFO:teuthology.orchestra.run.smithi103.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:08.326 INFO:teuthology.orchestra.run.smithi103.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T17:55:08.326 INFO:teuthology.orchestra.run.smithi103.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:08.326 INFO:teuthology.orchestra.run.smithi103.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T17:55:08.326 INFO:teuthology.orchestra.run.smithi103.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T17:55:08.326 INFO:teuthology.orchestra.run.smithi103.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T17:55:08.326 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:08.326 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:08.326 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:08.326 INFO:teuthology.orchestra.run.smithi103.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T17:55:08.326 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:08.326 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:08.327 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:08.327 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-25T17:55:08.327 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-25T17:55:08.327 INFO:teuthology.orchestra.run.smithi103.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-25T17:55:08.327 INFO:teuthology.orchestra.run.smithi103.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T17:55:08.327 INFO:teuthology.orchestra.run.smithi103.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T17:55:08.327 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T17:55:08.327 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-25T17:55:08.608 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-test 2024-03-25T17:55:09.160 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:15 ago on Mon 25 Mar 2024 05:53:54 PM UTC. 2024-03-25T17:55:09.317 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-25T17:55:09.317 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T17:55:09.317 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-03-25T17:55:09.318 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T17:55:09.318 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-25T17:55:09.318 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T17:55:09.318 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-03-25T17:55:09.318 INFO:teuthology.orchestra.run.smithi103.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T17:55:09.318 INFO:teuthology.orchestra.run.smithi103.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T17:55:09.318 INFO:teuthology.orchestra.run.smithi103.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T17:55:09.318 INFO:teuthology.orchestra.run.smithi103.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T17:55:09.318 INFO:teuthology.orchestra.run.smithi103.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T17:55:09.318 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T17:55:09.318 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-25T17:55:09.318 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T17:55:09.318 INFO:teuthology.orchestra.run.smithi103.stdout:Install 6 Packages 2024-03-25T17:55:09.319 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T17:55:09.319 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 48 M 2024-03-25T17:55:09.319 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 204 M 2024-03-25T17:55:09.319 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-25T17:55:09.654 INFO:teuthology.orchestra.run.smithi103.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-25T17:55:09.704 INFO:teuthology.orchestra.run.smithi103.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-25T17:55:09.737 INFO:teuthology.orchestra.run.smithi103.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 63 kB 00:00 2024-03-25T17:55:09.946 INFO:teuthology.orchestra.run.smithi103.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 901 kB/s | 187 kB 00:00 2024-03-25T17:55:10.413 INFO:teuthology.orchestra.run.smithi103.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 176 kB/s | 163 kB 00:00 2024-03-25T17:55:10.414 INFO:teuthology.orchestra.run.smithi083.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 341 kB/s | 747 kB 00:02 2024-03-25T17:55:10.414 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:55:10.415 INFO:teuthology.orchestra.run.smithi083.stdout:Total 341 kB/s | 747 kB 00:02 2024-03-25T17:55:10.415 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T17:55:10.421 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T17:55:10.421 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T17:55:10.427 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T17:55:10.428 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T17:55:10.501 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T17:55:10.927 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T17:55:11.028 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T17:55:11.408 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T17:55:11.699 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T17:55:11.699 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:55:11.699 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T17:55:11.700 INFO:teuthology.orchestra.run.smithi083.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:55:11.700 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:55:11.700 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T17:55:11.910 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T17:55:12.475 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:05:20 ago on Mon 25 Mar 2024 05:49:52 PM UTC. 2024-03-25T17:55:12.626 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T17:55:12.626 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T17:55:12.626 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-25T17:55:12.626 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T17:55:12.626 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T17:55:12.627 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T17:55:12.627 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:55:12.627 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T17:55:12.627 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T17:55:12.627 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T17:55:12.627 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:55:12.628 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 145 k 2024-03-25T17:55:12.628 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 436 k 2024-03-25T17:55:12.628 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T17:55:13.287 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 220 kB/s | 145 kB 00:00 2024-03-25T17:55:13.288 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:55:13.288 INFO:teuthology.orchestra.run.smithi083.stdout:Total 220 kB/s | 145 kB 00:00 2024-03-25T17:55:13.288 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T17:55:13.296 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T17:55:13.297 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T17:55:13.350 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T17:55:13.351 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T17:55:13.497 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T17:55:13.555 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T17:55:13.579 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T17:55:13.579 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:55:13.579 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T17:55:13.579 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:55:14.540 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T17:55:14.541 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:55:14.541 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T17:55:14.541 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:14.541 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:55:14.541 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T17:55:14.795 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mgr 2024-03-25T17:55:15.362 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:05:23 ago on Mon 25 Mar 2024 05:49:52 PM UTC. 2024-03-25T17:55:15.459 INFO:teuthology.orchestra.run.smithi083.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:55:15.511 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T17:55:15.512 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-03-25T17:55:15.513 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T17:55:15.571 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T17:55:16.126 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:05:24 ago on Mon 25 Mar 2024 05:49:52 PM UTC. 2024-03-25T17:55:16.274 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T17:55:16.275 INFO:teuthology.orchestra.run.smithi083.stdout:====================================================================================== 2024-03-25T17:55:16.276 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-03-25T17:55:16.276 INFO:teuthology.orchestra.run.smithi083.stdout:====================================================================================== 2024-03-25T17:55:16.276 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T17:55:16.276 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T17:55:16.276 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-03-25T17:55:16.276 INFO:teuthology.orchestra.run.smithi083.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T17:55:16.276 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T17:55:16.276 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T17:55:16.276 INFO:teuthology.orchestra.run.smithi083.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T17:55:16.276 INFO:teuthology.orchestra.run.smithi083.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T17:55:16.276 INFO:teuthology.orchestra.run.smithi083.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T17:55:16.276 INFO:teuthology.orchestra.run.smithi083.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T17:55:16.277 INFO:teuthology.orchestra.run.smithi083.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T17:55:16.277 INFO:teuthology.orchestra.run.smithi083.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T17:55:16.277 INFO:teuthology.orchestra.run.smithi083.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T17:55:16.277 INFO:teuthology.orchestra.run.smithi083.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T17:55:16.277 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:55:16.277 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T17:55:16.277 INFO:teuthology.orchestra.run.smithi083.stdout:====================================================================================== 2024-03-25T17:55:16.277 INFO:teuthology.orchestra.run.smithi083.stdout:Install 12 Packages 2024-03-25T17:55:16.277 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:55:16.278 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 6.9 M 2024-03-25T17:55:16.278 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 26 M 2024-03-25T17:55:16.278 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T17:55:17.000 INFO:teuthology.orchestra.run.smithi083.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 32 kB/s | 15 kB 00:00 2024-03-25T17:55:17.142 INFO:teuthology.orchestra.run.smithi083.stdout:(2/12): protobuf-3.14.0-13.el9.x86_64.rpm 7.1 MB/s | 1.0 MB 00:00 2024-03-25T17:55:17.184 INFO:teuthology.orchestra.run.smithi083.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 5.6 MB/s | 238 kB 00:00 2024-03-25T17:55:17.292 INFO:teuthology.orchestra.run.smithi083.stdout:(4/12): ceph-grafana-dashboards-19.0.0-1439.g5d 41 kB/s | 31 kB 00:00 2024-03-25T17:55:17.376 INFO:teuthology.orchestra.run.smithi083.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.4 MB/s | 863 kB 00:00 2024-03-25T17:55:17.452 INFO:teuthology.orchestra.run.smithi083.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 259 kB/s | 19 kB 00:00 2024-03-25T17:55:17.502 INFO:teuthology.orchestra.run.smithi083.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.6 MB/s | 548 kB 00:00 2024-03-25T17:55:17.552 INFO:teuthology.orchestra.run.smithi083.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.8 MB/s | 144 kB 00:00 2024-03-25T17:55:17.594 INFO:teuthology.orchestra.run.smithi083.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 737 kB/s | 31 kB 00:00 2024-03-25T17:55:17.636 INFO:teuthology.orchestra.run.smithi083.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 4.4 MB/s | 188 kB 00:00 2024-03-25T17:55:17.844 INFO:teuthology.orchestra.run.smithi083.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 5.2 MB/s | 2.0 MB 00:00 2024-03-25T17:55:19.911 INFO:teuthology.orchestra.run.smithi083.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 538 kB/s | 1.8 MB 00:03 2024-03-25T17:55:19.915 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:55:19.915 INFO:teuthology.orchestra.run.smithi083.stdout:Total 1.9 MB/s | 6.9 MB 00:03 2024-03-25T17:55:19.935 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T17:55:19.955 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T17:55:19.955 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T17:55:20.137 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T17:55:20.137 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T17:55:20.662 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T17:55:20.737 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T17:55:20.772 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T17:55:20.819 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T17:55:20.857 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T17:55:20.961 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T17:55:21.091 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T17:55:21.185 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T17:55:21.229 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T17:55:21.258 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T17:55:21.289 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T17:55:21.529 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T17:55:21.564 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T17:55:22.415 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T17:55:22.416 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T17:55:22.416 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T17:55:22.416 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T17:55:22.416 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T17:55:22.416 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T17:55:22.416 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T17:55:22.416 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T17:55:22.416 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T17:55:22.416 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T17:55:22.416 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T17:55:22.416 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T17:55:22.764 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T17:55:22.764 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:55:22.764 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T17:55:22.765 INFO:teuthology.orchestra.run.smithi083.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T17:55:22.765 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:55:22.765 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:55:22.765 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:55:22.765 INFO:teuthology.orchestra.run.smithi083.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T17:55:22.765 INFO:teuthology.orchestra.run.smithi083.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T17:55:22.765 INFO:teuthology.orchestra.run.smithi083.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T17:55:22.765 INFO:teuthology.orchestra.run.smithi083.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T17:55:22.765 INFO:teuthology.orchestra.run.smithi083.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T17:55:22.765 INFO:teuthology.orchestra.run.smithi083.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T17:55:22.765 INFO:teuthology.orchestra.run.smithi083.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T17:55:22.765 INFO:teuthology.orchestra.run.smithi083.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T17:55:22.765 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:55:22.765 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T17:55:23.013 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T17:55:23.573 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:05:31 ago on Mon 25 Mar 2024 05:49:52 PM UTC. 2024-03-25T17:55:23.725 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T17:55:23.726 INFO:teuthology.orchestra.run.smithi083.stdout:============================================================================================ 2024-03-25T17:55:23.726 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-03-25T17:55:23.726 INFO:teuthology.orchestra.run.smithi083.stdout:============================================================================================ 2024-03-25T17:55:23.726 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T17:55:23.726 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T17:55:23.726 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-03-25T17:55:23.727 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T17:55:23.727 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T17:55:23.727 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T17:55:23.727 INFO:teuthology.orchestra.run.smithi083.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T17:55:23.727 INFO:teuthology.orchestra.run.smithi083.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T17:55:23.727 INFO:teuthology.orchestra.run.smithi083.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T17:55:23.727 INFO:teuthology.orchestra.run.smithi083.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T17:55:23.727 INFO:teuthology.orchestra.run.smithi083.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T17:55:23.727 INFO:teuthology.orchestra.run.smithi083.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T17:55:23.727 INFO:teuthology.orchestra.run.smithi083.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T17:55:23.727 INFO:teuthology.orchestra.run.smithi083.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T17:55:23.727 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:55:23.727 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T17:55:23.728 INFO:teuthology.orchestra.run.smithi083.stdout:============================================================================================ 2024-03-25T17:55:23.728 INFO:teuthology.orchestra.run.smithi083.stdout:Install 12 Packages 2024-03-25T17:55:23.728 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:55:23.728 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 37 M 2024-03-25T17:55:23.728 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 209 M 2024-03-25T17:55:23.728 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T17:55:23.968 INFO:teuthology.orchestra.run.smithi083.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 261 kB/s | 32 kB 00:00 2024-03-25T17:55:24.027 INFO:teuthology.orchestra.run.smithi083.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 489 kB/s | 17 kB 00:00 2024-03-25T17:55:24.060 INFO:teuthology.orchestra.run.smithi083.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-25T17:55:24.236 INFO:teuthology.orchestra.run.smithi083.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 27 MB/s | 4.7 MB 00:00 2024-03-25T17:55:24.402 INFO:teuthology.orchestra.run.smithi083.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 29 MB/s | 4.8 MB 00:00 2024-03-25T17:55:24.445 INFO:teuthology.orchestra.run.smithi083.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.2 MB/s | 217 kB 00:00 2024-03-25T17:55:24.495 INFO:teuthology.orchestra.run.smithi083.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.6 MB/s | 3.0 MB 00:00 2024-03-25T17:55:24.912 INFO:teuthology.orchestra.run.smithi083.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 34 MB/s | 16 MB 00:00 2024-03-25T17:55:25.012 INFO:teuthology.orchestra.run.smithi083.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.5 MB/s | 803 kB 00:00 2024-03-25T17:55:25.121 INFO:teuthology.orchestra.run.smithi083.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 937 kB/s | 195 kB 00:00 2024-03-25T17:55:25.196 INFO:teuthology.orchestra.run.smithi083.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.3 MB/s | 245 kB 00:00 2024-03-25T17:55:40.111 INFO:teuthology.orchestra.run.smithi083.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 465 kB/s | 7.4 MB 00:16 2024-03-25T17:55:40.113 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:55:40.114 INFO:teuthology.orchestra.run.smithi083.stdout:Total 2.3 MB/s | 37 MB 00:16 2024-03-25T17:55:40.158 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T17:55:40.183 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T17:55:40.184 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T17:55:40.545 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T17:55:40.546 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T17:55:41.245 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T17:55:41.309 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T17:55:41.340 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T17:55:41.432 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T17:55:41.485 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T17:55:41.663 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T17:55:41.709 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T17:55:41.795 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T17:55:42.429 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T17:55:42.537 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T17:55:43.773 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T17:55:44.790 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T17:55:44.818 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T17:55:47.416 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T17:55:47.416 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T17:55:47.416 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T17:55:47.416 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T17:55:47.416 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T17:55:47.416 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T17:55:47.416 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T17:55:47.417 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T17:55:47.417 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T17:55:47.417 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T17:55:47.417 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T17:55:47.417 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T17:55:47.898 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T17:55:47.898 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:55:47.898 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T17:55:47.898 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:55:47.898 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T17:55:47.899 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T17:55:47.899 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T17:55:47.899 INFO:teuthology.orchestra.run.smithi083.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T17:55:47.899 INFO:teuthology.orchestra.run.smithi083.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T17:55:47.899 INFO:teuthology.orchestra.run.smithi083.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T17:55:47.899 INFO:teuthology.orchestra.run.smithi083.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T17:55:47.899 INFO:teuthology.orchestra.run.smithi083.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T17:55:47.899 INFO:teuthology.orchestra.run.smithi083.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T17:55:47.899 INFO:teuthology.orchestra.run.smithi083.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T17:55:47.899 INFO:teuthology.orchestra.run.smithi083.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T17:55:47.899 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:55:47.899 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T17:55:48.263 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mgr-rook 2024-03-25T17:55:48.836 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:05:56 ago on Mon 25 Mar 2024 05:49:52 PM UTC. 2024-03-25T17:55:48.985 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T17:55:48.986 INFO:teuthology.orchestra.run.smithi083.stdout:======================================================================================== 2024-03-25T17:55:48.986 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-03-25T17:55:48.986 INFO:teuthology.orchestra.run.smithi083.stdout:======================================================================================== 2024-03-25T17:55:48.987 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T17:55:48.987 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T17:55:48.987 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-03-25T17:55:48.987 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T17:55:48.987 INFO:teuthology.orchestra.run.smithi083.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T17:55:48.987 INFO:teuthology.orchestra.run.smithi083.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T17:55:48.987 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T17:55:48.987 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T17:55:48.987 INFO:teuthology.orchestra.run.smithi083.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T17:55:48.987 INFO:teuthology.orchestra.run.smithi083.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T17:55:48.987 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T17:55:48.988 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T17:55:48.988 INFO:teuthology.orchestra.run.smithi083.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T17:55:48.988 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T17:55:48.988 INFO:teuthology.orchestra.run.smithi083.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T17:55:48.988 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:55:48.988 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T17:55:48.988 INFO:teuthology.orchestra.run.smithi083.stdout:======================================================================================== 2024-03-25T17:55:48.988 INFO:teuthology.orchestra.run.smithi083.stdout:Install 13 Packages 2024-03-25T17:55:48.988 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:55:48.989 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 2.0 M 2024-03-25T17:55:48.989 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 26 M 2024-03-25T17:55:48.989 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T17:55:49.294 INFO:teuthology.orchestra.run.smithi083.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 138 kB/s | 18 kB 00:00 2024-03-25T17:55:49.319 INFO:teuthology.orchestra.run.smithi083.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 156 kB/s | 24 kB 00:00 2024-03-25T17:55:49.412 INFO:teuthology.orchestra.run.smithi083.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.5 MB/s | 174 kB 00:00 2024-03-25T17:55:49.437 INFO:teuthology.orchestra.run.smithi083.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.1 MB/s | 132 kB 00:00 2024-03-25T17:55:49.479 INFO:teuthology.orchestra.run.smithi083.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.1 MB/s | 211 kB 00:00 2024-03-25T17:55:49.504 INFO:teuthology.orchestra.run.smithi083.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 641 kB/s | 43 kB 00:00 2024-03-25T17:55:49.754 INFO:teuthology.orchestra.run.smithi083.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 88 kB/s | 52 kB 00:00 2024-03-25T17:55:49.780 INFO:teuthology.orchestra.run.smithi083.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 54 kB/s | 15 kB 00:00 2024-03-25T17:55:49.805 INFO:teuthology.orchestra.run.smithi083.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 99 kB/s | 32 kB 00:00 2024-03-25T17:55:50.055 INFO:teuthology.orchestra.run.smithi083.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 674 kB/s | 203 kB 00:00 2024-03-25T17:55:50.139 INFO:teuthology.orchestra.run.smithi083.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 1.1 MB/s | 90 kB 00:00 2024-03-25T17:55:50.164 INFO:teuthology.orchestra.run.smithi083.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 165 kB/s | 59 kB 00:00 2024-03-25T17:55:50.222 INFO:teuthology.orchestra.run.smithi083.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.3 MB/s | 1.0 MB 00:00 2024-03-25T17:55:50.226 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:55:50.226 INFO:teuthology.orchestra.run.smithi083.stdout:Total 1.7 MB/s | 2.0 MB 00:01 2024-03-25T17:55:50.227 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T17:55:50.239 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T17:55:50.239 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T17:55:50.330 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T17:55:50.330 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T17:55:50.873 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T17:55:51.025 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T17:55:51.089 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T17:55:51.153 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T17:55:51.183 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T17:55:51.224 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T17:55:51.317 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T17:55:51.427 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T17:55:51.475 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T17:55:52.185 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T17:55:52.240 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T17:55:52.273 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T17:55:52.324 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T17:55:52.346 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T17:55:53.175 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T17:55:53.175 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T17:55:53.175 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T17:55:53.175 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T17:55:53.176 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T17:55:53.176 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T17:55:53.176 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T17:55:53.176 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T17:55:53.176 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T17:55:53.176 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T17:55:53.176 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T17:55:53.176 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T17:55:53.176 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T17:55:53.535 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T17:55:53.535 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:55:53.535 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T17:55:53.535 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:55:53.535 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T17:55:53.535 INFO:teuthology.orchestra.run.smithi083.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T17:55:53.535 INFO:teuthology.orchestra.run.smithi083.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T17:55:53.535 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T17:55:53.536 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T17:55:53.536 INFO:teuthology.orchestra.run.smithi083.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T17:55:53.536 INFO:teuthology.orchestra.run.smithi083.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T17:55:53.536 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T17:55:53.536 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T17:55:53.536 INFO:teuthology.orchestra.run.smithi083.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T17:55:53.536 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T17:55:53.536 INFO:teuthology.orchestra.run.smithi083.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T17:55:53.536 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:55:53.536 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T17:55:53.781 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T17:55:54.337 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:06:02 ago on Mon 25 Mar 2024 05:49:52 PM UTC. 2024-03-25T17:55:54.487 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T17:55:54.488 INFO:teuthology.orchestra.run.smithi083.stdout:======================================================================================== 2024-03-25T17:55:54.488 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-03-25T17:55:54.488 INFO:teuthology.orchestra.run.smithi083.stdout:======================================================================================== 2024-03-25T17:55:54.488 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T17:55:54.489 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T17:55:54.489 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-03-25T17:55:54.489 INFO:teuthology.orchestra.run.smithi083.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T17:55:54.489 INFO:teuthology.orchestra.run.smithi083.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T17:55:54.489 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T17:55:54.489 INFO:teuthology.orchestra.run.smithi083.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T17:55:54.489 INFO:teuthology.orchestra.run.smithi083.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T17:55:54.489 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:55:54.489 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T17:55:54.489 INFO:teuthology.orchestra.run.smithi083.stdout:======================================================================================== 2024-03-25T17:55:54.489 INFO:teuthology.orchestra.run.smithi083.stdout:Install 6 Packages 2024-03-25T17:55:54.489 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:55:54.489 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 6.8 M 2024-03-25T17:55:54.490 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 33 M 2024-03-25T17:55:54.490 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T17:55:54.778 INFO:teuthology.orchestra.run.smithi083.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.4 MB/s | 248 kB 00:00 2024-03-25T17:55:54.894 INFO:teuthology.orchestra.run.smithi083.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 4.7 MB/s | 548 kB 00:00 2024-03-25T17:55:54.970 INFO:teuthology.orchestra.run.smithi083.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 5.8 MB 00:00 2024-03-25T17:55:55.003 INFO:teuthology.orchestra.run.smithi083.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 533 kB/s | 58 kB 00:00 2024-03-25T17:55:55.062 INFO:teuthology.orchestra.run.smithi083.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 553 kB/s | 51 kB 00:00 2024-03-25T17:55:55.329 INFO:teuthology.orchestra.run.smithi083.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 189 kB/s | 138 kB 00:00 2024-03-25T17:55:55.330 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:55:55.331 INFO:teuthology.orchestra.run.smithi083.stdout:Total 8.1 MB/s | 6.8 MB 00:00 2024-03-25T17:55:55.344 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T17:55:55.354 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T17:55:55.354 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T17:55:55.442 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T17:55:55.444 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T17:55:55.744 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T17:55:55.928 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T17:55:55.989 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T17:55:56.511 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T17:55:56.600 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T17:55:56.670 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T17:55:56.693 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T17:55:57.475 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T17:55:57.475 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T17:55:57.476 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T17:55:57.476 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T17:55:57.476 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T17:55:57.476 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T17:55:57.778 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T17:55:57.778 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:55:57.778 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T17:55:57.778 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:55:57.778 INFO:teuthology.orchestra.run.smithi083.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T17:55:57.778 INFO:teuthology.orchestra.run.smithi083.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T17:55:57.779 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T17:55:57.779 INFO:teuthology.orchestra.run.smithi083.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T17:55:57.779 INFO:teuthology.orchestra.run.smithi083.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T17:55:57.779 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:55:57.779 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T17:55:58.029 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-fuse 2024-03-25T17:55:58.590 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:06:06 ago on Mon 25 Mar 2024 05:49:52 PM UTC. 2024-03-25T17:55:58.740 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T17:55:58.740 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T17:55:58.741 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-03-25T17:55:58.741 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T17:55:58.741 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T17:55:58.741 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T17:55:58.741 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-03-25T17:55:58.741 INFO:teuthology.orchestra.run.smithi083.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T17:55:58.741 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:55:58.741 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T17:55:58.741 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T17:55:58.741 INFO:teuthology.orchestra.run.smithi083.stdout:Install 2 Packages 2024-03-25T17:55:58.741 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:55:58.742 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 964 k 2024-03-25T17:55:58.742 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 2.8 M 2024-03-25T17:55:58.742 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T17:55:58.856 INFO:teuthology.orchestra.run.smithi083.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 692 kB/s | 79 kB 00:00 2024-03-25T17:56:01.748 INFO:teuthology.orchestra.run.smithi083.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 294 kB/s | 885 kB 00:03 2024-03-25T17:56:01.749 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:56:01.749 INFO:teuthology.orchestra.run.smithi083.stdout:Total 320 kB/s | 964 kB 00:03 2024-03-25T17:56:01.750 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T17:56:01.774 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T17:56:01.774 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T17:56:01.852 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T17:56:01.852 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T17:56:02.033 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T17:56:02.165 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T17:56:02.241 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T17:56:02.915 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T17:56:02.916 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T17:56:03.233 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T17:56:03.233 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:56:03.233 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T17:56:03.233 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T17:56:03.233 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:56:03.233 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T17:56:03.516 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-volume 2024-03-25T17:56:04.091 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:06:12 ago on Mon 25 Mar 2024 05:49:52 PM UTC. 2024-03-25T17:56:04.241 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T17:56:04.241 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T17:56:04.242 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-03-25T17:56:04.242 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T17:56:04.242 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T17:56:04.242 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T17:56:04.242 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-03-25T17:56:04.242 INFO:teuthology.orchestra.run.smithi083.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T17:56:04.242 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:56:04.242 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T17:56:04.242 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T17:56:04.242 INFO:teuthology.orchestra.run.smithi083.stdout:Install 2 Packages 2024-03-25T17:56:04.242 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:56:04.242 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 332 k 2024-03-25T17:56:04.242 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 1.5 M 2024-03-25T17:56:04.243 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T17:56:04.352 INFO:teuthology.orchestra.run.smithi083.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 629 kB/s | 69 kB 00:00 2024-03-25T17:56:05.803 INFO:teuthology.orchestra.run.smithi083.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 169 kB/s | 264 kB 00:01 2024-03-25T17:56:05.804 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:56:05.805 INFO:teuthology.orchestra.run.smithi083.stdout:Total 213 kB/s | 332 kB 00:01 2024-03-25T17:56:05.805 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T17:56:05.815 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T17:56:05.815 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T17:56:05.858 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T17:56:05.858 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T17:56:06.011 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T17:56:06.175 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T17:56:06.222 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T17:56:06.238 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T17:56:06.238 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:56:06.238 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T17:56:06.238 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:56:06.941 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T17:56:07.275 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T17:56:07.275 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:56:07.275 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T17:56:07.275 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:56:07.276 INFO:teuthology.orchestra.run.smithi083.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T17:56:07.276 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:56:07.276 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T17:56:07.597 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install librados-devel 2024-03-25T17:56:08.164 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:06:16 ago on Mon 25 Mar 2024 05:49:52 PM UTC. 2024-03-25T17:56:08.312 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T17:56:08.313 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T17:56:08.313 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-25T17:56:08.313 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T17:56:08.313 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T17:56:08.313 INFO:teuthology.orchestra.run.smithi083.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T17:56:08.313 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:56:08.314 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T17:56:08.314 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T17:56:08.314 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T17:56:08.314 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:56:08.314 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 127 k 2024-03-25T17:56:08.314 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 456 k 2024-03-25T17:56:08.314 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T17:56:09.959 INFO:teuthology.orchestra.run.smithi083.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 77 kB/s | 127 kB 00:01 2024-03-25T17:56:09.959 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:56:09.960 INFO:teuthology.orchestra.run.smithi083.stdout:Total 77 kB/s | 127 kB 00:01 2024-03-25T17:56:09.960 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T17:56:09.966 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T17:56:09.966 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T17:56:10.026 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T17:56:10.026 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T17:56:10.174 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T17:56:10.291 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:56:10.693 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:56:10.991 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:56:10.991 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:56:10.991 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T17:56:10.991 INFO:teuthology.orchestra.run.smithi083.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:56:10.991 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:56:10.991 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T17:56:11.235 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install libcephfs2 2024-03-25T17:56:11.809 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:06:19 ago on Mon 25 Mar 2024 05:49:52 PM UTC. 2024-03-25T17:56:11.910 INFO:teuthology.orchestra.run.smithi083.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:56:11.966 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T17:56:11.966 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-03-25T17:56:11.967 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T17:56:12.030 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install libcephfs-devel 2024-03-25T17:56:12.588 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:06:20 ago on Mon 25 Mar 2024 05:49:52 PM UTC. 2024-03-25T17:56:12.738 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T17:56:12.739 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T17:56:12.739 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-25T17:56:12.739 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T17:56:12.739 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T17:56:12.739 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T17:56:12.739 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:56:12.739 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T17:56:12.739 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T17:56:12.739 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T17:56:12.739 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:56:12.739 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 31 k 2024-03-25T17:56:12.740 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 140 k 2024-03-25T17:56:12.740 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T17:56:13.296 INFO:teuthology.orchestra.run.smithi083.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 56 kB/s | 31 kB 00:00 2024-03-25T17:56:13.297 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:56:13.297 INFO:teuthology.orchestra.run.smithi083.stdout:Total 56 kB/s | 31 kB 00:00 2024-03-25T17:56:13.297 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T17:56:13.300 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T17:56:13.301 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T17:56:13.327 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T17:56:13.327 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T17:56:13.452 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T17:56:13.544 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:56:13.911 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:56:14.182 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:56:14.182 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:56:14.183 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T17:56:14.183 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:56:14.183 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:56:14.183 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T17:56:14.373 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install librados2 2024-03-25T17:56:14.930 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:06:22 ago on Mon 25 Mar 2024 05:49:52 PM UTC. 2024-03-25T17:56:15.027 INFO:teuthology.orchestra.run.smithi083.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:56:15.077 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T17:56:15.078 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-03-25T17:56:15.078 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T17:56:15.129 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install librbd1 2024-03-25T17:56:15.709 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:06:23 ago on Mon 25 Mar 2024 05:49:52 PM UTC. 2024-03-25T17:56:15.816 INFO:teuthology.orchestra.run.smithi083.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:56:15.870 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T17:56:15.871 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-03-25T17:56:15.871 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T17:56:15.921 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install python3-rados 2024-03-25T17:56:16.503 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:06:24 ago on Mon 25 Mar 2024 05:49:52 PM UTC. 2024-03-25T17:56:16.604 INFO:teuthology.orchestra.run.smithi083.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:56:16.657 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T17:56:16.658 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-03-25T17:56:16.658 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T17:56:16.709 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install python3-rgw 2024-03-25T17:56:17.263 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:06:25 ago on Mon 25 Mar 2024 05:49:52 PM UTC. 2024-03-25T17:56:17.359 INFO:teuthology.orchestra.run.smithi083.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:56:17.409 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T17:56:17.410 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-03-25T17:56:17.410 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T17:56:17.457 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install python3-cephfs 2024-03-25T17:56:18.020 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:06:26 ago on Mon 25 Mar 2024 05:49:52 PM UTC. 2024-03-25T17:56:18.115 INFO:teuthology.orchestra.run.smithi083.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:56:18.168 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T17:56:18.169 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-03-25T17:56:18.169 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T17:56:18.227 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install python3-rbd 2024-03-25T17:56:18.780 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:06:26 ago on Mon 25 Mar 2024 05:49:52 PM UTC. 2024-03-25T17:56:18.876 INFO:teuthology.orchestra.run.smithi083.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:56:18.927 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T17:56:18.928 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-03-25T17:56:18.928 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T17:56:18.982 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install rbd-fuse 2024-03-25T17:56:19.537 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:06:27 ago on Mon 25 Mar 2024 05:49:52 PM UTC. 2024-03-25T17:56:19.684 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T17:56:19.685 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T17:56:19.685 INFO:teuthology.orchestra.run.smithi083.stdout: Package Architecture Version Repository Size 2024-03-25T17:56:19.685 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T17:56:19.685 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T17:56:19.685 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T17:56:19.685 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:56:19.685 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T17:56:19.685 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T17:56:19.685 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T17:56:19.685 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:56:19.685 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 85 k 2024-03-25T17:56:19.685 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 231 k 2024-03-25T17:56:19.686 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T17:56:20.291 INFO:teuthology.orchestra.run.smithi083.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 141 kB/s | 85 kB 00:00 2024-03-25T17:56:20.292 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:56:20.292 INFO:teuthology.orchestra.run.smithi083.stdout:Total 141 kB/s | 85 kB 00:00 2024-03-25T17:56:20.292 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T17:56:20.299 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T17:56:20.299 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T17:56:20.355 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T17:56:20.355 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T17:56:20.490 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T17:56:20.597 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:56:20.979 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:56:21.271 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:56:21.271 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:56:21.271 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T17:56:21.271 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:56:21.272 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:56:21.272 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T17:56:21.542 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install rbd-mirror 2024-03-25T17:56:22.114 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:06:30 ago on Mon 25 Mar 2024 05:49:52 PM UTC. 2024-03-25T17:56:22.267 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T17:56:22.267 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T17:56:22.267 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-25T17:56:22.267 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T17:56:22.267 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T17:56:22.267 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T17:56:22.267 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:56:22.267 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T17:56:22.267 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T17:56:22.268 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T17:56:22.268 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:56:22.268 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 3.1 M 2024-03-25T17:56:22.268 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 13 M 2024-03-25T17:56:22.268 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T17:56:35.285 INFO:teuthology.orchestra.run.smithi083.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 243 kB/s | 3.1 MB 00:13 2024-03-25T17:56:35.286 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:56:35.286 INFO:teuthology.orchestra.run.smithi083.stdout:Total 243 kB/s | 3.1 MB 00:13 2024-03-25T17:56:35.287 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T17:56:35.298 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T17:56:35.298 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T17:56:35.374 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T17:56:35.375 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T17:56:35.814 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T17:56:35.869 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:56:35.894 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:56:35.894 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:56:35.894 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T17:56:35.895 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T17:56:35.895 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T17:56:35.895 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:56:36.963 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:56:36.963 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:56:36.964 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T17:56:36.964 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:56:36.964 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:56:36.964 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T17:56:37.182 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install rbd-nbd 2024-03-25T17:56:37.757 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:06:45 ago on Mon 25 Mar 2024 05:49:52 PM UTC. 2024-03-25T17:56:37.906 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T17:56:37.907 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T17:56:37.907 INFO:teuthology.orchestra.run.smithi083.stdout: Package Architecture Version Repository Size 2024-03-25T17:56:37.907 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T17:56:37.907 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T17:56:37.907 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T17:56:37.907 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:56:37.907 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T17:56:37.907 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T17:56:37.907 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T17:56:37.907 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:56:37.907 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 171 k 2024-03-25T17:56:37.908 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 494 k 2024-03-25T17:56:37.908 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T17:56:38.989 INFO:teuthology.orchestra.run.smithi083.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 159 kB/s | 171 kB 00:01 2024-03-25T17:56:38.989 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:56:38.989 INFO:teuthology.orchestra.run.smithi083.stdout:Total 158 kB/s | 171 kB 00:01 2024-03-25T17:56:38.990 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T17:56:38.999 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T17:56:38.999 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T17:56:39.058 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T17:56:39.058 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T17:56:39.209 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T17:56:39.304 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:56:39.722 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:56:40.039 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:56:40.039 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:56:40.039 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T17:56:40.040 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:56:40.040 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:56:40.040 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T17:56:40.256 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:56:40.827 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:06:48 ago on Mon 25 Mar 2024 05:49:52 PM UTC. 2024-03-25T17:56:40.980 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T17:56:40.980 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T17:56:40.980 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-25T17:56:40.980 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T17:56:40.980 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T17:56:40.980 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T17:56:40.981 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-03-25T17:56:40.981 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T17:56:40.981 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:56:40.981 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T17:56:40.981 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T17:56:40.981 INFO:teuthology.orchestra.run.smithi083.stdout:Install 2 Packages 2024-03-25T17:56:40.981 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T17:56:40.982 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 91 M 2024-03-25T17:56:40.982 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 498 M 2024-03-25T17:56:40.982 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T17:57:53.660 INFO:teuthology.orchestra.run.smithi083.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 316 kB/s | 22 MB 01:12 2024-03-25T17:58:00.029 INFO:teuthology.orchestra.run.smithi103.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 284 kB/s | 47 MB 02:50 2024-03-25T17:58:00.031 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:58:00.032 INFO:teuthology.orchestra.run.smithi103.stdout:Total 289 kB/s | 48 MB 02:50 2024-03-25T17:58:00.043 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-25T17:58:00.062 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-25T17:58:00.062 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-25T17:58:00.445 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-25T17:58:00.445 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-25T17:58:01.048 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-25T17:58:01.129 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T17:58:01.171 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T17:58:01.212 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T17:58:01.263 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T17:58:01.290 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T17:58:05.757 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T17:58:05.832 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T17:58:08.154 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T17:58:08.154 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T17:58:08.154 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T17:58:08.154 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T17:58:08.154 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T17:58:08.155 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T17:58:08.537 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T17:58:08.537 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T17:58:08.537 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-25T17:58:08.537 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:58:08.537 INFO:teuthology.orchestra.run.smithi103.stdout: jq-1.6-16.el9.x86_64 2024-03-25T17:58:08.537 INFO:teuthology.orchestra.run.smithi103.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:58:08.537 INFO:teuthology.orchestra.run.smithi103.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T17:58:08.537 INFO:teuthology.orchestra.run.smithi103.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T17:58:08.538 INFO:teuthology.orchestra.run.smithi103.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T17:58:08.538 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T17:58:08.538 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-25T17:58:08.840 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph 2024-03-25T17:58:09.414 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:04:15 ago on Mon 25 Mar 2024 05:53:54 PM UTC. 2024-03-25T17:58:09.572 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-25T17:58:09.575 INFO:teuthology.orchestra.run.smithi103.stdout:========================================================================================= 2024-03-25T17:58:09.575 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-03-25T17:58:09.575 INFO:teuthology.orchestra.run.smithi103.stdout:========================================================================================= 2024-03-25T17:58:09.575 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-25T17:58:09.575 INFO:teuthology.orchestra.run.smithi103.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T17:58:09.575 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-03-25T17:58:09.575 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T17:58:09.576 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T17:58:09.576 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T17:58:09.576 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T17:58:09.576 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T17:58:09.576 INFO:teuthology.orchestra.run.smithi103.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T17:58:09.576 INFO:teuthology.orchestra.run.smithi103.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T17:58:09.576 INFO:teuthology.orchestra.run.smithi103.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T17:58:09.576 INFO:teuthology.orchestra.run.smithi103.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T17:58:09.576 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T17:58:09.576 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T17:58:09.576 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T17:58:09.576 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T17:58:09.577 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T17:58:09.577 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T17:58:09.577 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T17:58:09.577 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T17:58:09.577 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T17:58:09.577 INFO:teuthology.orchestra.run.smithi103.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T17:58:09.577 INFO:teuthology.orchestra.run.smithi103.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T17:58:09.577 INFO:teuthology.orchestra.run.smithi103.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T17:58:09.577 INFO:teuthology.orchestra.run.smithi103.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T17:58:09.577 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T17:58:09.577 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T17:58:09.577 INFO:teuthology.orchestra.run.smithi103.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T17:58:09.577 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T17:58:09.577 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T17:58:09.578 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T17:58:09.578 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T17:58:09.578 INFO:teuthology.orchestra.run.smithi103.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T17:58:09.578 INFO:teuthology.orchestra.run.smithi103.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T17:58:09.578 INFO:teuthology.orchestra.run.smithi103.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T17:58:09.578 INFO:teuthology.orchestra.run.smithi103.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T17:58:09.578 INFO:teuthology.orchestra.run.smithi103.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T17:58:09.578 INFO:teuthology.orchestra.run.smithi103.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T17:58:09.578 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T17:58:09.578 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-25T17:58:09.578 INFO:teuthology.orchestra.run.smithi103.stdout:========================================================================================= 2024-03-25T17:58:09.578 INFO:teuthology.orchestra.run.smithi103.stdout:Install 36 Packages 2024-03-25T17:58:09.578 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T17:58:09.579 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 30 M 2024-03-25T17:58:09.579 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 107 M 2024-03-25T17:58:09.579 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-25T17:58:10.342 INFO:teuthology.orchestra.run.smithi103.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 19 kB/s | 6.6 kB 00:00 2024-03-25T17:58:14.775 INFO:teuthology.orchestra.run.smithi103.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 319 kB/s | 1.5 MB 00:04 2024-03-25T17:58:14.825 INFO:teuthology.orchestra.run.smithi103.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 450 kB/s | 2.1 MB 00:04 2024-03-25T17:58:15.376 INFO:teuthology.orchestra.run.smithi103.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 450 kB/s | 247 kB 00:00 2024-03-25T17:58:15.501 INFO:teuthology.orchestra.run.smithi103.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-25T17:58:15.543 INFO:teuthology.orchestra.run.smithi103.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T17:58:15.576 INFO:teuthology.orchestra.run.smithi103.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 950 kB/s | 32 kB 00:00 2024-03-25T17:58:15.610 INFO:teuthology.orchestra.run.smithi103.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-25T17:58:15.644 INFO:teuthology.orchestra.run.smithi103.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-25T17:58:15.702 INFO:teuthology.orchestra.run.smithi103.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-03-25T17:58:15.769 INFO:teuthology.orchestra.run.smithi103.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-25T17:58:15.803 INFO:teuthology.orchestra.run.smithi103.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-25T17:58:15.836 INFO:teuthology.orchestra.run.smithi103.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.7 MB/s | 124 kB 00:00 2024-03-25T17:58:15.870 INFO:teuthology.orchestra.run.smithi103.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-25T17:58:15.987 INFO:teuthology.orchestra.run.smithi103.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 1.0 MB/s | 125 kB 00:00 2024-03-25T17:58:16.071 INFO:teuthology.orchestra.run.smithi103.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 268 kB/s | 22 kB 00:00 2024-03-25T17:58:16.563 INFO:teuthology.orchestra.run.smithi103.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 307 kB/s | 151 kB 00:00 2024-03-25T17:58:16.688 INFO:teuthology.orchestra.run.smithi103.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 348 kB/s | 43 kB 00:00 2024-03-25T17:58:17.014 INFO:teuthology.orchestra.run.smithi103.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 528 kB/s | 172 kB 00:00 2024-03-25T17:58:17.281 INFO:teuthology.orchestra.run.smithi103.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-03-25T17:58:17.439 INFO:teuthology.orchestra.run.smithi103.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 67 kB/s | 11 kB 00:00 2024-03-25T17:58:17.514 INFO:teuthology.orchestra.run.smithi103.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 235 kB/s | 18 kB 00:00 2024-03-25T17:58:17.765 INFO:teuthology.orchestra.run.smithi103.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 93 kB/s | 23 kB 00:00 2024-03-25T17:58:17.857 INFO:teuthology.orchestra.run.smithi103.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 212 kB/s | 19 kB 00:00 2024-03-25T17:58:17.924 INFO:teuthology.orchestra.run.smithi103.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 295 kB/s | 20 kB 00:00 2024-03-25T17:58:18.074 INFO:teuthology.orchestra.run.smithi103.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 309 kB/s | 46 kB 00:00 2024-03-25T17:58:18.166 INFO:teuthology.orchestra.run.smithi103.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 860 kB/s | 79 kB 00:00 2024-03-25T17:58:18.292 INFO:teuthology.orchestra.run.smithi103.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2024-03-25T17:58:18.484 INFO:teuthology.orchestra.run.smithi103.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 86 kB/s | 16 kB 00:00 2024-03-25T17:58:18.568 INFO:teuthology.orchestra.run.smithi103.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.1 MB/s | 90 kB 00:00 2024-03-25T17:58:18.676 INFO:teuthology.orchestra.run.smithi103.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 331 kB/s | 36 kB 00:00 2024-03-25T17:58:18.818 INFO:teuthology.orchestra.run.smithi103.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.6 MB/s | 230 kB 00:00 2024-03-25T17:58:18.952 INFO:teuthology.orchestra.run.smithi103.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.1 MB/s | 427 kB 00:00 2024-03-25T17:58:19.052 INFO:teuthology.orchestra.run.smithi103.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 199 kB/s | 20 kB 00:00 2024-03-25T17:58:23.160 INFO:teuthology.orchestra.run.smithi103.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 374 kB/s | 4.7 MB 00:12 2024-03-25T17:59:03.982 INFO:teuthology.orchestra.run.smithi103.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 349 kB/s | 17 MB 00:49 2024-03-25T17:59:03.986 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:59:03.986 INFO:teuthology.orchestra.run.smithi103.stdout:Total 558 kB/s | 30 MB 00:54 2024-03-25T17:59:04.008 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-25T17:59:04.050 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-25T17:59:04.050 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-25T17:59:04.453 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-25T17:59:04.453 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-25T17:59:05.805 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-25T17:59:05.861 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T17:59:05.899 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T17:59:05.928 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T17:59:05.999 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T17:59:06.044 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T17:59:06.076 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T17:59:06.168 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T17:59:06.263 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T17:59:06.305 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T17:59:06.376 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T17:59:06.430 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T17:59:06.469 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T17:59:06.526 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T17:59:06.581 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T17:59:06.767 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T17:59:06.815 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T17:59:06.845 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T17:59:06.881 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T17:59:06.910 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T17:59:06.987 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T17:59:07.020 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T17:59:07.054 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T17:59:07.094 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T17:59:07.129 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T17:59:07.164 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T17:59:07.254 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T17:59:07.294 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T17:59:07.366 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T17:59:07.502 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T17:59:07.628 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T17:59:07.802 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T17:59:07.832 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T17:59:07.858 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T17:59:07.858 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:59:07.858 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T17:59:07.858 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T17:59:07.858 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T17:59:07.858 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T17:59:09.292 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T17:59:09.318 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T17:59:09.318 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:59:09.318 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T17:59:09.318 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T17:59:09.318 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T17:59:09.318 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T17:59:09.751 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T17:59:09.773 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T17:59:09.773 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:59:09.773 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T17:59:09.773 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T17:59:09.774 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T17:59:09.774 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T17:59:09.977 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T17:59:09.999 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T17:59:09.999 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:59:10.000 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T17:59:10.000 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T17:59:10.000 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T17:59:10.000 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T17:59:10.066 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T17:59:11.669 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T17:59:11.669 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T17:59:11.669 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T17:59:11.669 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T17:59:11.670 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T17:59:11.670 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T17:59:11.670 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T17:59:11.670 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T17:59:11.670 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T17:59:11.670 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T17:59:11.671 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T17:59:11.671 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T17:59:11.671 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T17:59:11.671 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T17:59:11.671 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T17:59:11.671 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T17:59:11.671 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T17:59:11.671 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T17:59:11.671 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T17:59:11.671 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T17:59:11.671 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T17:59:11.671 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T17:59:11.672 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T17:59:11.672 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T17:59:11.672 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T17:59:11.672 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T17:59:11.672 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T17:59:11.672 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T17:59:11.672 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T17:59:11.673 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T17:59:11.673 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T17:59:11.673 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T17:59:11.673 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T17:59:11.673 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T17:59:11.674 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T17:59:11.674 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T17:59:12.795 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T17:59:12.795 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T17:59:12.795 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-25T17:59:12.795 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:59:12.795 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:59:12.795 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:59:12.796 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:59:12.796 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:59:12.796 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:59:12.796 INFO:teuthology.orchestra.run.smithi103.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T17:59:12.796 INFO:teuthology.orchestra.run.smithi103.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T17:59:12.796 INFO:teuthology.orchestra.run.smithi103.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T17:59:12.796 INFO:teuthology.orchestra.run.smithi103.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T17:59:12.796 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T17:59:12.796 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T17:59:12.796 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T17:59:12.796 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T17:59:12.796 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T17:59:12.796 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T17:59:12.796 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T17:59:12.796 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T17:59:12.797 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T17:59:12.797 INFO:teuthology.orchestra.run.smithi103.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T17:59:12.797 INFO:teuthology.orchestra.run.smithi103.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T17:59:12.797 INFO:teuthology.orchestra.run.smithi103.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T17:59:12.797 INFO:teuthology.orchestra.run.smithi103.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T17:59:12.797 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T17:59:12.797 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T17:59:12.797 INFO:teuthology.orchestra.run.smithi103.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T17:59:12.797 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T17:59:12.797 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T17:59:12.797 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T17:59:12.797 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T17:59:12.797 INFO:teuthology.orchestra.run.smithi103.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T17:59:12.798 INFO:teuthology.orchestra.run.smithi103.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T17:59:12.798 INFO:teuthology.orchestra.run.smithi103.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T17:59:12.798 INFO:teuthology.orchestra.run.smithi103.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T17:59:12.798 INFO:teuthology.orchestra.run.smithi103.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T17:59:12.798 INFO:teuthology.orchestra.run.smithi103.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T17:59:12.798 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T17:59:12.798 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-25T17:59:13.221 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-base 2024-03-25T17:59:13.795 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:05:19 ago on Mon 25 Mar 2024 05:53:54 PM UTC. 2024-03-25T17:59:13.891 INFO:teuthology.orchestra.run.smithi103.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:59:13.943 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-25T17:59:13.944 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-03-25T17:59:13.944 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-25T17:59:14.018 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install cephadm 2024-03-25T17:59:14.577 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:05:20 ago on Mon 25 Mar 2024 05:53:54 PM UTC. 2024-03-25T17:59:14.726 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-25T17:59:14.727 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T17:59:14.728 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-03-25T17:59:14.728 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T17:59:14.728 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-25T17:59:14.728 INFO:teuthology.orchestra.run.smithi103.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T17:59:14.728 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T17:59:14.728 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-25T17:59:14.728 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T17:59:14.728 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-03-25T17:59:14.728 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T17:59:14.728 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 747 k 2024-03-25T17:59:14.728 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 754 k 2024-03-25T17:59:14.728 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-25T17:59:18.695 INFO:teuthology.orchestra.run.smithi103.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 188 kB/s | 747 kB 00:03 2024-03-25T17:59:18.696 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:59:18.696 INFO:teuthology.orchestra.run.smithi103.stdout:Total 188 kB/s | 747 kB 00:03 2024-03-25T17:59:18.696 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-25T17:59:18.702 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-25T17:59:18.703 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-25T17:59:18.709 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-25T17:59:18.709 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-25T17:59:18.789 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-25T17:59:19.190 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T17:59:19.325 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T17:59:19.701 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T17:59:19.979 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T17:59:19.979 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T17:59:19.979 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-25T17:59:19.979 INFO:teuthology.orchestra.run.smithi103.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:59:19.979 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T17:59:19.980 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-25T17:59:20.209 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T17:59:20.776 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:05:26 ago on Mon 25 Mar 2024 05:53:54 PM UTC. 2024-03-25T17:59:20.928 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-25T17:59:20.929 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T17:59:20.929 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-03-25T17:59:20.929 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T17:59:20.930 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-25T17:59:20.930 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T17:59:20.930 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T17:59:20.930 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-25T17:59:20.930 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T17:59:20.930 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-03-25T17:59:20.931 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T17:59:20.931 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 145 k 2024-03-25T17:59:20.931 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 436 k 2024-03-25T17:59:20.931 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-25T17:59:21.728 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 182 kB/s | 145 kB 00:00 2024-03-25T17:59:21.728 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:59:21.729 INFO:teuthology.orchestra.run.smithi103.stdout:Total 182 kB/s | 145 kB 00:00 2024-03-25T17:59:21.729 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-25T17:59:21.737 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-25T17:59:21.737 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-25T17:59:21.791 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-25T17:59:21.792 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-25T17:59:21.933 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-25T17:59:22.004 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T17:59:22.028 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T17:59:22.029 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:59:22.029 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T17:59:22.029 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T17:59:23.012 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T17:59:23.012 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T17:59:23.012 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-25T17:59:23.012 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:59:23.012 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T17:59:23.012 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-25T17:59:23.273 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mgr 2024-03-25T17:59:23.847 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:05:29 ago on Mon 25 Mar 2024 05:53:54 PM UTC. 2024-03-25T17:59:23.945 INFO:teuthology.orchestra.run.smithi103.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:59:23.996 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-25T17:59:23.997 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-03-25T17:59:23.997 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-25T17:59:24.081 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T17:59:24.636 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:05:30 ago on Mon 25 Mar 2024 05:53:54 PM UTC. 2024-03-25T17:59:24.784 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-25T17:59:24.785 INFO:teuthology.orchestra.run.smithi103.stdout:====================================================================================== 2024-03-25T17:59:24.785 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-03-25T17:59:24.785 INFO:teuthology.orchestra.run.smithi103.stdout:====================================================================================== 2024-03-25T17:59:24.785 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-25T17:59:24.785 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T17:59:24.785 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-03-25T17:59:24.785 INFO:teuthology.orchestra.run.smithi103.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T17:59:24.785 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T17:59:24.785 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T17:59:24.786 INFO:teuthology.orchestra.run.smithi103.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T17:59:24.786 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T17:59:24.786 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T17:59:24.786 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T17:59:24.786 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T17:59:24.786 INFO:teuthology.orchestra.run.smithi103.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T17:59:24.786 INFO:teuthology.orchestra.run.smithi103.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T17:59:24.786 INFO:teuthology.orchestra.run.smithi103.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T17:59:24.786 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T17:59:24.786 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-25T17:59:24.786 INFO:teuthology.orchestra.run.smithi103.stdout:====================================================================================== 2024-03-25T17:59:24.786 INFO:teuthology.orchestra.run.smithi103.stdout:Install 12 Packages 2024-03-25T17:59:24.786 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T17:59:24.787 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 6.9 M 2024-03-25T17:59:24.787 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 26 M 2024-03-25T17:59:24.787 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-25T17:59:25.434 INFO:teuthology.orchestra.run.smithi103.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 72 kB/s | 31 kB 00:00 2024-03-25T17:59:25.617 INFO:teuthology.orchestra.run.smithi103.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 25 kB/s | 15 kB 00:00 2024-03-25T17:59:25.717 INFO:teuthology.orchestra.run.smithi103.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.6 MB/s | 1.0 MB 00:00 2024-03-25T17:59:25.743 INFO:teuthology.orchestra.run.smithi103.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.9 MB/s | 238 kB 00:00 2024-03-25T17:59:26.376 INFO:teuthology.orchestra.run.smithi103.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.3 MB/s | 863 kB 00:00 2024-03-25T17:59:26.402 INFO:teuthology.orchestra.run.smithi103.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 832 kB/s | 548 kB 00:00 2024-03-25T17:59:26.644 INFO:teuthology.orchestra.run.smithi103.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 73 kB/s | 19 kB 00:00 2024-03-25T17:59:26.694 INFO:teuthology.orchestra.run.smithi103.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 7.0 MB/s | 2.0 MB 00:00 2024-03-25T17:59:26.827 INFO:teuthology.orchestra.run.smithi103.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 231 kB/s | 31 kB 00:00 2024-03-25T17:59:26.936 INFO:teuthology.orchestra.run.smithi103.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 494 kB/s | 144 kB 00:00 2024-03-25T17:59:26.961 INFO:teuthology.orchestra.run.smithi103.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.4 MB/s | 188 kB 00:00 2024-03-25T17:59:30.228 INFO:teuthology.orchestra.run.smithi103.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 349 kB/s | 1.8 MB 00:05 2024-03-25T17:59:30.231 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:59:30.232 INFO:teuthology.orchestra.run.smithi103.stdout:Total 1.3 MB/s | 6.9 MB 00:05 2024-03-25T17:59:30.252 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-25T17:59:30.270 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-25T17:59:30.271 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-25T17:59:30.463 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-25T17:59:30.464 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-25T17:59:30.966 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-25T17:59:31.037 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T17:59:31.081 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T17:59:31.126 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T17:59:31.166 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T17:59:31.267 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T17:59:31.401 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T17:59:31.487 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T17:59:31.538 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T17:59:31.567 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T17:59:31.598 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T17:59:31.840 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T17:59:31.873 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T17:59:32.713 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T17:59:32.713 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T17:59:32.713 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T17:59:32.713 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T17:59:32.714 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T17:59:32.714 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T17:59:32.714 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T17:59:32.714 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T17:59:32.714 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T17:59:32.714 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T17:59:32.714 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T17:59:32.714 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T17:59:33.093 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T17:59:33.093 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T17:59:33.093 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-25T17:59:33.093 INFO:teuthology.orchestra.run.smithi103.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T17:59:33.093 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:59:33.093 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:59:33.093 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:59:33.094 INFO:teuthology.orchestra.run.smithi103.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T17:59:33.094 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T17:59:33.094 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T17:59:33.094 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T17:59:33.094 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T17:59:33.094 INFO:teuthology.orchestra.run.smithi103.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T17:59:33.094 INFO:teuthology.orchestra.run.smithi103.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T17:59:33.094 INFO:teuthology.orchestra.run.smithi103.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T17:59:33.094 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T17:59:33.094 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-25T17:59:33.367 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T17:59:33.924 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:05:39 ago on Mon 25 Mar 2024 05:53:54 PM UTC. 2024-03-25T17:59:34.077 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-25T17:59:34.078 INFO:teuthology.orchestra.run.smithi103.stdout:============================================================================================ 2024-03-25T17:59:34.078 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-03-25T17:59:34.078 INFO:teuthology.orchestra.run.smithi103.stdout:============================================================================================ 2024-03-25T17:59:34.078 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-25T17:59:34.078 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T17:59:34.079 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-03-25T17:59:34.079 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T17:59:34.079 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T17:59:34.079 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T17:59:34.079 INFO:teuthology.orchestra.run.smithi103.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T17:59:34.079 INFO:teuthology.orchestra.run.smithi103.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T17:59:34.079 INFO:teuthology.orchestra.run.smithi103.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T17:59:34.079 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T17:59:34.079 INFO:teuthology.orchestra.run.smithi103.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T17:59:34.079 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T17:59:34.079 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T17:59:34.079 INFO:teuthology.orchestra.run.smithi103.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T17:59:34.079 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T17:59:34.079 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-25T17:59:34.080 INFO:teuthology.orchestra.run.smithi103.stdout:============================================================================================ 2024-03-25T17:59:34.080 INFO:teuthology.orchestra.run.smithi103.stdout:Install 12 Packages 2024-03-25T17:59:34.080 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T17:59:34.080 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 37 M 2024-03-25T17:59:34.080 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 209 M 2024-03-25T17:59:34.080 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-25T17:59:34.295 INFO:teuthology.orchestra.run.smithi103.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 271 kB/s | 32 kB 00:00 2024-03-25T17:59:34.353 INFO:teuthology.orchestra.run.smithi103.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 484 kB/s | 17 kB 00:00 2024-03-25T17:59:34.387 INFO:teuthology.orchestra.run.smithi103.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-25T17:59:34.579 INFO:teuthology.orchestra.run.smithi103.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 25 MB/s | 4.7 MB 00:00 2024-03-25T17:59:34.721 INFO:teuthology.orchestra.run.smithi103.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 4.8 MB 00:00 2024-03-25T17:59:34.763 INFO:teuthology.orchestra.run.smithi103.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.3 MB/s | 217 kB 00:00 2024-03-25T17:59:34.829 INFO:teuthology.orchestra.run.smithi103.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.6 MB/s | 3.0 MB 00:00 2024-03-25T17:59:35.122 INFO:teuthology.orchestra.run.smithi103.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 44 MB/s | 16 MB 00:00 2024-03-25T17:59:35.397 INFO:teuthology.orchestra.run.smithi103.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 711 kB/s | 195 kB 00:00 2024-03-25T17:59:35.430 INFO:teuthology.orchestra.run.smithi103.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 803 kB 00:00 2024-03-25T17:59:35.614 INFO:teuthology.orchestra.run.smithi103.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.1 MB/s | 245 kB 00:00 2024-03-25T17:59:44.456 INFO:teuthology.orchestra.run.smithi103.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 736 kB/s | 7.4 MB 00:10 2024-03-25T17:59:44.459 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:59:44.460 INFO:teuthology.orchestra.run.smithi103.stdout:Total 3.6 MB/s | 37 MB 00:10 2024-03-25T17:59:44.503 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-25T17:59:44.529 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-25T17:59:44.529 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-25T17:59:44.895 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-25T17:59:44.896 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-25T17:59:45.604 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-25T17:59:45.670 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T17:59:45.701 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T17:59:45.796 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T17:59:45.830 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T17:59:46.008 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T17:59:46.037 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T17:59:46.122 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T17:59:46.797 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T17:59:46.917 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T17:59:48.173 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T17:59:49.214 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T17:59:49.247 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T17:59:51.762 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T17:59:51.762 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T17:59:51.763 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T17:59:51.763 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T17:59:51.763 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T17:59:51.763 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T17:59:51.763 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T17:59:51.763 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T17:59:51.763 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T17:59:51.763 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T17:59:51.763 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T17:59:51.763 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T17:59:52.212 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T17:59:52.212 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T17:59:52.212 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-25T17:59:52.212 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:59:52.212 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T17:59:52.212 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T17:59:52.212 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T17:59:52.213 INFO:teuthology.orchestra.run.smithi103.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T17:59:52.213 INFO:teuthology.orchestra.run.smithi103.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T17:59:52.213 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T17:59:52.213 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T17:59:52.213 INFO:teuthology.orchestra.run.smithi103.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T17:59:52.213 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T17:59:52.213 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T17:59:52.213 INFO:teuthology.orchestra.run.smithi103.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T17:59:52.213 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T17:59:52.213 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-25T17:59:52.596 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mgr-rook 2024-03-25T17:59:53.170 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:05:59 ago on Mon 25 Mar 2024 05:53:54 PM UTC. 2024-03-25T17:59:53.321 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-25T17:59:53.322 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================== 2024-03-25T17:59:53.322 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-03-25T17:59:53.322 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================== 2024-03-25T17:59:53.322 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-25T17:59:53.323 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T17:59:53.323 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-03-25T17:59:53.323 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T17:59:53.323 INFO:teuthology.orchestra.run.smithi103.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T17:59:53.323 INFO:teuthology.orchestra.run.smithi103.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T17:59:53.323 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T17:59:53.323 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T17:59:53.323 INFO:teuthology.orchestra.run.smithi103.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T17:59:53.323 INFO:teuthology.orchestra.run.smithi103.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T17:59:53.323 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T17:59:53.323 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T17:59:53.323 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T17:59:53.323 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T17:59:53.323 INFO:teuthology.orchestra.run.smithi103.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T17:59:53.324 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T17:59:53.324 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-25T17:59:53.324 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================== 2024-03-25T17:59:53.324 INFO:teuthology.orchestra.run.smithi103.stdout:Install 13 Packages 2024-03-25T17:59:53.324 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T17:59:53.324 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 2.0 M 2024-03-25T17:59:53.324 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 26 M 2024-03-25T17:59:53.325 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-25T17:59:53.568 INFO:teuthology.orchestra.run.smithi103.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 267 kB/s | 24 kB 00:00 2024-03-25T17:59:53.610 INFO:teuthology.orchestra.run.smithi103.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 136 kB/s | 18 kB 00:00 2024-03-25T17:59:53.652 INFO:teuthology.orchestra.run.smithi103.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.8 MB/s | 174 kB 00:00 2024-03-25T17:59:53.677 INFO:teuthology.orchestra.run.smithi103.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-25T17:59:53.702 INFO:teuthology.orchestra.run.smithi103.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T17:59:53.719 INFO:teuthology.orchestra.run.smithi103.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-03-25T17:59:53.845 INFO:teuthology.orchestra.run.smithi103.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 120 kB/s | 15 kB 00:00 2024-03-25T17:59:53.861 INFO:teuthology.orchestra.run.smithi103.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 203 kB/s | 32 kB 00:00 2024-03-25T17:59:54.003 INFO:teuthology.orchestra.run.smithi103.stdout:(9/13): python3-google-auth-2.28.2-1.el9.noarch 1.2 MB/s | 203 kB 00:00 2024-03-25T17:59:54.104 INFO:teuthology.orchestra.run.smithi103.stdout:(10/13): python3-kubernetes-26.1.0-2.el9.noarch 4.7 MB/s | 1.0 MB 00:00 2024-03-25T17:59:54.121 INFO:teuthology.orchestra.run.smithi103.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 506 kB/s | 59 kB 00:00 2024-03-25T17:59:54.221 INFO:teuthology.orchestra.run.smithi103.stdout:(12/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el 70 kB/s | 52 kB 00:00 2024-03-25T17:59:54.246 INFO:teuthology.orchestra.run.smithi103.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 633 kB/s | 90 kB 00:00 2024-03-25T17:59:54.248 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:59:54.249 INFO:teuthology.orchestra.run.smithi103.stdout:Total 2.2 MB/s | 2.0 MB 00:00 2024-03-25T17:59:54.249 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-25T17:59:54.262 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-25T17:59:54.262 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-25T17:59:54.353 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-25T17:59:54.353 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-25T17:59:54.858 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-25T17:59:55.027 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T17:59:55.089 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T17:59:55.145 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T17:59:55.182 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T17:59:55.224 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T17:59:55.336 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T17:59:55.453 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T17:59:55.509 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T17:59:56.243 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T17:59:56.290 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T17:59:56.324 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T17:59:56.372 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T17:59:56.394 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T17:59:57.250 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T17:59:57.250 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T17:59:57.251 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T17:59:57.251 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T17:59:57.251 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T17:59:57.251 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T17:59:57.251 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T17:59:57.251 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T17:59:57.252 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T17:59:57.252 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T17:59:57.252 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T17:59:57.252 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T17:59:57.252 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T17:59:57.602 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T17:59:57.602 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T17:59:57.602 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-25T17:59:57.603 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:59:57.603 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T17:59:57.603 INFO:teuthology.orchestra.run.smithi103.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T17:59:57.603 INFO:teuthology.orchestra.run.smithi103.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T17:59:57.603 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T17:59:57.603 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T17:59:57.603 INFO:teuthology.orchestra.run.smithi103.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T17:59:57.603 INFO:teuthology.orchestra.run.smithi103.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T17:59:57.603 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T17:59:57.603 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T17:59:57.603 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T17:59:57.603 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T17:59:57.603 INFO:teuthology.orchestra.run.smithi103.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T17:59:57.603 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T17:59:57.604 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-25T17:59:57.897 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T17:59:58.458 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:06:04 ago on Mon 25 Mar 2024 05:53:54 PM UTC. 2024-03-25T17:59:58.608 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-25T17:59:58.609 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================== 2024-03-25T17:59:58.609 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-03-25T17:59:58.609 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================== 2024-03-25T17:59:58.610 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-25T17:59:58.610 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T17:59:58.610 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-03-25T17:59:58.610 INFO:teuthology.orchestra.run.smithi103.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T17:59:58.610 INFO:teuthology.orchestra.run.smithi103.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T17:59:58.610 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T17:59:58.611 INFO:teuthology.orchestra.run.smithi103.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T17:59:58.611 INFO:teuthology.orchestra.run.smithi103.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T17:59:58.611 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T17:59:58.611 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-25T17:59:58.611 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================== 2024-03-25T17:59:58.612 INFO:teuthology.orchestra.run.smithi103.stdout:Install 6 Packages 2024-03-25T17:59:58.612 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T17:59:58.613 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 6.8 M 2024-03-25T17:59:58.613 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 33 M 2024-03-25T17:59:58.613 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-25T17:59:58.959 INFO:teuthology.orchestra.run.smithi103.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-25T17:59:59.543 INFO:teuthology.orchestra.run.smithi103.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 162 kB/s | 138 kB 00:00 2024-03-25T17:59:59.668 INFO:teuthology.orchestra.run.smithi103.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 801 kB/s | 548 kB 00:00 2024-03-25T17:59:59.693 INFO:teuthology.orchestra.run.smithi103.stdout:(4/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 248 kB/s | 248 kB 00:00 2024-03-25T17:59:59.768 INFO:teuthology.orchestra.run.smithi103.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 509 kB/s | 51 kB 00:00 2024-03-25T17:59:59.969 INFO:teuthology.orchestra.run.smithi103.stdout:(6/6): python3-natsort-7.1.1-5.el9.noarch.rpm 136 kB/s | 58 kB 00:00 2024-03-25T17:59:59.972 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:59:59.972 INFO:teuthology.orchestra.run.smithi103.stdout:Total 5.0 MB/s | 6.8 MB 00:01 2024-03-25T17:59:59.985 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-25T17:59:59.996 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-25T17:59:59.996 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-25T18:00:00.086 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-25T18:00:00.087 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-25T18:00:00.383 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-25T18:00:00.562 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T18:00:00.612 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T18:00:01.140 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T18:00:01.216 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T18:00:01.285 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T18:00:01.316 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:00:02.136 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:00:02.136 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T18:00:02.136 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T18:00:02.136 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T18:00:02.136 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T18:00:02.136 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T18:00:02.431 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T18:00:02.431 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:00:02.431 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-25T18:00:02.431 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:00:02.431 INFO:teuthology.orchestra.run.smithi103.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T18:00:02.431 INFO:teuthology.orchestra.run.smithi103.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T18:00:02.431 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T18:00:02.431 INFO:teuthology.orchestra.run.smithi103.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T18:00:02.431 INFO:teuthology.orchestra.run.smithi103.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T18:00:02.431 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:00:02.431 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-25T18:00:02.691 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-fuse 2024-03-25T18:00:03.255 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:06:09 ago on Mon 25 Mar 2024 05:53:54 PM UTC. 2024-03-25T18:00:03.405 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-25T18:00:03.406 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T18:00:03.406 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-03-25T18:00:03.406 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T18:00:03.406 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-25T18:00:03.406 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T18:00:03.406 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-03-25T18:00:03.406 INFO:teuthology.orchestra.run.smithi103.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T18:00:03.407 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:00:03.407 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-25T18:00:03.407 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T18:00:03.407 INFO:teuthology.orchestra.run.smithi103.stdout:Install 2 Packages 2024-03-25T18:00:03.407 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:00:03.407 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 964 k 2024-03-25T18:00:03.407 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 2.8 M 2024-03-25T18:00:03.408 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-25T18:00:03.530 INFO:teuthology.orchestra.run.smithi103.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 645 kB/s | 79 kB 00:00 2024-03-25T18:00:06.896 INFO:teuthology.orchestra.run.smithi103.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 254 kB/s | 885 kB 00:03 2024-03-25T18:00:06.898 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:00:06.898 INFO:teuthology.orchestra.run.smithi103.stdout:Total 276 kB/s | 964 kB 00:03 2024-03-25T18:00:06.898 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-25T18:00:06.922 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-25T18:00:06.922 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-25T18:00:06.998 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-25T18:00:06.999 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-25T18:00:07.196 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-25T18:00:07.330 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T18:00:07.404 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:00:08.078 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:00:08.078 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:00:08.391 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T18:00:08.391 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:00:08.391 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-25T18:00:08.391 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T18:00:08.391 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:00:08.391 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-25T18:00:08.713 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-volume 2024-03-25T18:00:09.284 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:06:15 ago on Mon 25 Mar 2024 05:53:54 PM UTC. 2024-03-25T18:00:09.438 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-25T18:00:09.439 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T18:00:09.439 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-03-25T18:00:09.439 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T18:00:09.439 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-25T18:00:09.439 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T18:00:09.439 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-03-25T18:00:09.439 INFO:teuthology.orchestra.run.smithi103.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T18:00:09.439 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:00:09.439 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-25T18:00:09.439 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T18:00:09.439 INFO:teuthology.orchestra.run.smithi103.stdout:Install 2 Packages 2024-03-25T18:00:09.440 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:00:09.440 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 332 k 2024-03-25T18:00:09.440 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 1.5 M 2024-03-25T18:00:09.440 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-25T18:00:09.552 INFO:teuthology.orchestra.run.smithi103.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 612 kB/s | 69 kB 00:00 2024-03-25T18:00:11.627 INFO:teuthology.orchestra.run.smithi103.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 121 kB/s | 264 kB 00:02 2024-03-25T18:00:11.629 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:00:11.629 INFO:teuthology.orchestra.run.smithi103.stdout:Total 152 kB/s | 332 kB 00:02 2024-03-25T18:00:11.629 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-25T18:00:11.639 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-25T18:00:11.639 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-25T18:00:11.682 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-25T18:00:11.682 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-25T18:00:11.832 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-25T18:00:12.010 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T18:00:12.055 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:00:12.070 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:00:12.070 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:00:12.070 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T18:00:12.071 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:00:12.787 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T18:00:13.121 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T18:00:13.121 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:00:13.121 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-25T18:00:13.121 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:00:13.122 INFO:teuthology.orchestra.run.smithi103.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T18:00:13.122 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:00:13.122 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-25T18:00:13.467 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install librados-devel 2024-03-25T18:00:14.041 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:06:20 ago on Mon 25 Mar 2024 05:53:54 PM UTC. 2024-03-25T18:00:14.189 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-25T18:00:14.190 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T18:00:14.190 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-03-25T18:00:14.190 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T18:00:14.190 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-25T18:00:14.190 INFO:teuthology.orchestra.run.smithi103.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T18:00:14.190 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:00:14.190 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-25T18:00:14.191 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T18:00:14.191 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-03-25T18:00:14.191 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:00:14.191 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 127 k 2024-03-25T18:00:14.191 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 456 k 2024-03-25T18:00:14.191 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-25T18:00:15.127 INFO:teuthology.orchestra.run.smithi103.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 135 kB/s | 127 kB 00:00 2024-03-25T18:00:15.127 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:00:15.128 INFO:teuthology.orchestra.run.smithi103.stdout:Total 135 kB/s | 127 kB 00:00 2024-03-25T18:00:15.128 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-25T18:00:15.133 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-25T18:00:15.134 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-25T18:00:15.191 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-25T18:00:15.192 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-25T18:00:15.336 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-25T18:00:15.450 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:00:15.850 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:00:16.146 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:00:16.146 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:00:16.146 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-25T18:00:16.146 INFO:teuthology.orchestra.run.smithi103.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:00:16.146 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:00:16.146 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-25T18:00:16.395 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install libcephfs2 2024-03-25T18:00:16.972 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:06:22 ago on Mon 25 Mar 2024 05:53:54 PM UTC. 2024-03-25T18:00:17.068 INFO:teuthology.orchestra.run.smithi103.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:00:17.121 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-25T18:00:17.122 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-03-25T18:00:17.122 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-25T18:00:17.203 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install libcephfs-devel 2024-03-25T18:00:17.766 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:06:23 ago on Mon 25 Mar 2024 05:53:54 PM UTC. 2024-03-25T18:00:17.914 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-25T18:00:17.915 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T18:00:17.915 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-03-25T18:00:17.915 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T18:00:17.915 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-25T18:00:17.915 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T18:00:17.915 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:00:17.916 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-25T18:00:17.916 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T18:00:17.916 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-03-25T18:00:17.916 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:00:17.916 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 31 k 2024-03-25T18:00:17.916 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 140 k 2024-03-25T18:00:17.916 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-25T18:00:18.639 INFO:teuthology.orchestra.run.smithi103.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 43 kB/s | 31 kB 00:00 2024-03-25T18:00:18.639 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:00:18.640 INFO:teuthology.orchestra.run.smithi103.stdout:Total 43 kB/s | 31 kB 00:00 2024-03-25T18:00:18.640 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-25T18:00:18.644 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-25T18:00:18.644 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-25T18:00:18.672 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-25T18:00:18.673 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-25T18:00:18.816 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-25T18:00:18.930 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:00:19.305 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:00:19.561 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:00:19.562 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:00:19.562 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-25T18:00:19.562 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:00:19.562 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:00:19.562 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-25T18:00:19.719 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install librados2 2024-03-25T18:00:20.285 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:06:26 ago on Mon 25 Mar 2024 05:53:54 PM UTC. 2024-03-25T18:00:20.381 INFO:teuthology.orchestra.run.smithi103.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:00:20.432 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-25T18:00:20.433 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-03-25T18:00:20.433 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-25T18:00:20.482 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install librbd1 2024-03-25T18:00:21.079 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:06:27 ago on Mon 25 Mar 2024 05:53:54 PM UTC. 2024-03-25T18:00:21.181 INFO:teuthology.orchestra.run.smithi103.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:00:21.235 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-25T18:00:21.236 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-03-25T18:00:21.236 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-25T18:00:21.283 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install python3-rados 2024-03-25T18:00:21.862 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:06:27 ago on Mon 25 Mar 2024 05:53:54 PM UTC. 2024-03-25T18:00:21.957 INFO:teuthology.orchestra.run.smithi103.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:00:22.008 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-25T18:00:22.009 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-03-25T18:00:22.009 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-25T18:00:22.051 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install python3-rgw 2024-03-25T18:00:22.613 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:06:28 ago on Mon 25 Mar 2024 05:53:54 PM UTC. 2024-03-25T18:00:22.709 INFO:teuthology.orchestra.run.smithi103.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:00:22.761 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-25T18:00:22.762 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-03-25T18:00:22.762 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-25T18:00:22.810 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install python3-cephfs 2024-03-25T18:00:23.369 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:06:29 ago on Mon 25 Mar 2024 05:53:54 PM UTC. 2024-03-25T18:00:23.464 INFO:teuthology.orchestra.run.smithi103.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:00:23.515 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-25T18:00:23.516 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-03-25T18:00:23.517 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-25T18:00:23.564 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install python3-rbd 2024-03-25T18:00:24.122 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:06:30 ago on Mon 25 Mar 2024 05:53:54 PM UTC. 2024-03-25T18:00:24.219 INFO:teuthology.orchestra.run.smithi103.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:00:24.271 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-25T18:00:24.272 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-03-25T18:00:24.272 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-25T18:00:24.321 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install rbd-fuse 2024-03-25T18:00:24.881 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:06:30 ago on Mon 25 Mar 2024 05:53:54 PM UTC. 2024-03-25T18:00:25.028 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-25T18:00:25.028 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T18:00:25.029 INFO:teuthology.orchestra.run.smithi103.stdout: Package Architecture Version Repository Size 2024-03-25T18:00:25.029 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T18:00:25.029 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-25T18:00:25.029 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T18:00:25.029 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:00:25.029 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-25T18:00:25.029 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T18:00:25.029 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-03-25T18:00:25.029 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:00:25.029 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 85 k 2024-03-25T18:00:25.029 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 231 k 2024-03-25T18:00:25.029 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-25T18:00:25.943 INFO:teuthology.orchestra.run.smithi103.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 93 kB/s | 85 kB 00:00 2024-03-25T18:00:25.944 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:00:25.944 INFO:teuthology.orchestra.run.smithi103.stdout:Total 93 kB/s | 85 kB 00:00 2024-03-25T18:00:25.944 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-25T18:00:25.953 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-25T18:00:25.953 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-25T18:00:26.012 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-25T18:00:26.013 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-25T18:00:26.149 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-25T18:00:26.241 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:00:26.610 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:00:26.914 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:00:26.914 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:00:26.914 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-25T18:00:26.914 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:00:26.914 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:00:26.914 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-25T18:00:27.138 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install rbd-mirror 2024-03-25T18:00:27.708 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:06:33 ago on Mon 25 Mar 2024 05:53:54 PM UTC. 2024-03-25T18:00:27.862 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-25T18:00:27.863 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T18:00:27.863 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-03-25T18:00:27.863 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T18:00:27.863 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-25T18:00:27.863 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:00:27.864 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:00:27.864 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-25T18:00:27.864 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T18:00:27.864 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-03-25T18:00:27.864 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:00:27.864 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 3.1 M 2024-03-25T18:00:27.864 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 13 M 2024-03-25T18:00:27.864 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-25T18:00:38.064 INFO:teuthology.orchestra.run.smithi103.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 310 kB/s | 3.1 MB 00:10 2024-03-25T18:00:38.064 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:00:38.065 INFO:teuthology.orchestra.run.smithi103.stdout:Total 310 kB/s | 3.1 MB 00:10 2024-03-25T18:00:38.065 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-25T18:00:38.076 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-25T18:00:38.076 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-25T18:00:38.153 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-25T18:00:38.153 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-25T18:00:38.579 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-25T18:00:38.656 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:00:38.680 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:00:38.680 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:00:38.680 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T18:00:38.680 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:00:38.680 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:00:38.680 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:00:39.752 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:00:39.752 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:00:39.752 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-25T18:00:39.752 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:00:39.752 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:00:39.752 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-25T18:00:39.998 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install rbd-nbd 2024-03-25T18:00:40.571 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:06:46 ago on Mon 25 Mar 2024 05:53:54 PM UTC. 2024-03-25T18:00:40.720 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-25T18:00:40.721 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T18:00:40.721 INFO:teuthology.orchestra.run.smithi103.stdout: Package Architecture Version Repository Size 2024-03-25T18:00:40.721 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T18:00:40.721 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-25T18:00:40.721 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T18:00:40.721 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:00:40.721 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-25T18:00:40.721 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T18:00:40.722 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-03-25T18:00:40.722 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:00:40.722 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 171 k 2024-03-25T18:00:40.722 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 494 k 2024-03-25T18:00:40.722 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-25T18:00:41.718 INFO:teuthology.orchestra.run.smithi103.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 172 kB/s | 171 kB 00:00 2024-03-25T18:00:41.719 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:00:41.720 INFO:teuthology.orchestra.run.smithi103.stdout:Total 172 kB/s | 171 kB 00:00 2024-03-25T18:00:41.720 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-25T18:00:41.729 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-25T18:00:41.729 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-25T18:00:41.790 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-25T18:00:41.790 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-25T18:00:41.950 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-25T18:00:42.060 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:00:42.487 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:00:42.767 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:00:42.767 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:00:42.767 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-25T18:00:42.767 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:00:42.767 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:00:42.767 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-25T18:00:43.019 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-base-debuginfo 2024-03-25T18:00:43.584 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:06:49 ago on Mon 25 Mar 2024 05:53:54 PM UTC. 2024-03-25T18:00:43.737 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-25T18:00:43.737 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T18:00:43.737 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-03-25T18:00:43.737 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T18:00:43.737 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-25T18:00:43.738 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T18:00:43.738 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-03-25T18:00:43.738 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T18:00:43.738 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:00:43.738 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-25T18:00:43.738 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T18:00:43.738 INFO:teuthology.orchestra.run.smithi103.stdout:Install 2 Packages 2024-03-25T18:00:43.738 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:00:43.739 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 91 M 2024-03-25T18:00:43.739 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 498 M 2024-03-25T18:00:43.739 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-25T18:00:59.973 INFO:teuthology.orchestra.run.smithi083.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 269 kB/s | 68 MB 04:18 2024-03-25T18:00:59.974 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:00:59.975 INFO:teuthology.orchestra.run.smithi083.stdout:Total 358 kB/s | 91 MB 04:18 2024-03-25T18:00:59.975 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T18:00:59.980 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T18:00:59.980 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T18:01:00.560 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T18:01:00.560 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T18:01:03.653 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T18:01:10.866 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:01:10.941 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:01:14.343 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:01:14.344 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T18:01:14.625 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:01:14.625 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T18:01:14.625 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T18:01:14.625 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:14.626 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:14.626 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T18:01:14.626 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T18:01:14.848 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-common-debuginfo 2024-03-25T18:01:15.410 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:11:23 ago on Mon 25 Mar 2024 05:49:52 PM UTC. 2024-03-25T18:01:15.559 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T18:01:15.560 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T18:01:15.560 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-25T18:01:15.560 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T18:01:15.560 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T18:01:15.560 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T18:01:15.560 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T18:01:15.560 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T18:01:15.560 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T18:01:15.560 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T18:01:15.560 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T18:01:15.560 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 416 M 2024-03-25T18:01:15.560 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 1.7 G 2024-03-25T18:01:15.561 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T18:02:15.490 INFO:teuthology.orchestra.run.smithi103.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 251 kB/s | 22 MB 01:31 2024-03-25T18:04:17.561 INFO:teuthology.orchestra.run.smithi103.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 326 kB/s | 68 MB 03:33 2024-03-25T18:04:17.562 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:17.562 INFO:teuthology.orchestra.run.smithi103.stdout:Total 434 kB/s | 91 MB 03:33 2024-03-25T18:04:17.562 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-25T18:04:17.566 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-25T18:04:17.566 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-25T18:04:18.161 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-25T18:04:18.161 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-25T18:04:21.283 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-25T18:04:28.480 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:04:28.645 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:04:32.194 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:04:32.195 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T18:04:32.454 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:04:32.455 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:04:32.455 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-25T18:04:32.455 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:32.455 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:32.455 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:04:32.455 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-25T18:04:32.710 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-common-debuginfo 2024-03-25T18:04:33.275 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:10:39 ago on Mon 25 Mar 2024 05:53:54 PM UTC. 2024-03-25T18:04:33.424 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-25T18:04:33.424 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T18:04:33.424 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-03-25T18:04:33.424 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T18:04:33.424 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-25T18:04:33.424 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T18:04:33.424 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:04:33.425 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-25T18:04:33.425 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T18:04:33.425 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-03-25T18:04:33.425 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:04:33.425 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 416 M 2024-03-25T18:04:33.425 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 1.7 G 2024-03-25T18:04:33.425 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-25T18:20:19.914 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 372 kB/s | 416 MB 19:04 2024-03-25T18:20:19.915 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:20:19.915 INFO:teuthology.orchestra.run.smithi083.stdout:Total 372 kB/s | 416 MB 19:04 2024-03-25T18:20:19.915 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T18:20:19.918 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T18:20:19.918 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T18:20:22.379 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T18:20:22.379 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T18:21:05.060 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T18:21:05.469 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:21:13.938 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:21:14.203 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:21:14.203 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T18:21:14.203 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T18:21:14.203 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:21:14.203 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T18:21:14.203 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T18:21:14.464 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T18:21:15.028 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:31:23 ago on Mon 25 Mar 2024 05:49:52 PM UTC. 2024-03-25T18:21:15.175 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T18:21:15.175 INFO:teuthology.orchestra.run.smithi083.stdout:======================================================================================== 2024-03-25T18:21:15.175 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-25T18:21:15.175 INFO:teuthology.orchestra.run.smithi083.stdout:======================================================================================== 2024-03-25T18:21:15.176 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T18:21:15.176 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T18:21:15.176 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T18:21:15.176 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T18:21:15.176 INFO:teuthology.orchestra.run.smithi083.stdout:======================================================================================== 2024-03-25T18:21:15.176 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T18:21:15.176 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T18:21:15.176 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 1.4 M 2024-03-25T18:21:15.176 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 6.4 M 2024-03-25T18:21:15.176 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T18:21:20.038 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 301 kB/s | 1.4 MB 00:04 2024-03-25T18:21:20.039 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:21:20.039 INFO:teuthology.orchestra.run.smithi083.stdout:Total 300 kB/s | 1.4 MB 00:04 2024-03-25T18:21:20.039 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T18:21:20.044 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T18:21:20.044 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T18:21:20.088 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T18:21:20.089 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T18:21:20.392 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T18:21:20.494 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:21:20.916 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:21:21.190 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:21:21.190 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T18:21:21.190 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T18:21:21.190 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:21:21.190 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T18:21:21.191 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T18:21:21.360 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T18:21:21.917 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:31:29 ago on Mon 25 Mar 2024 05:49:52 PM UTC. 2024-03-25T18:21:22.064 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T18:21:22.065 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T18:21:22.065 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-25T18:21:22.065 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T18:21:22.065 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T18:21:22.065 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T18:21:22.065 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T18:21:22.065 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T18:21:22.066 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T18:21:22.066 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T18:21:22.066 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T18:21:22.066 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 176 M 2024-03-25T18:21:22.066 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 753 M 2024-03-25T18:21:22.066 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T18:24:55.127 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 349 kB/s | 416 MB 20:21 2024-03-25T18:24:55.128 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:24:55.128 INFO:teuthology.orchestra.run.smithi103.stdout:Total 349 kB/s | 416 MB 20:21 2024-03-25T18:24:55.128 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-25T18:24:55.131 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-25T18:24:55.131 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-25T18:24:57.609 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-25T18:24:57.609 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-25T18:25:06.790 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 800 kB/s | 176 MB 03:44 2024-03-25T18:25:06.791 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:25:06.791 INFO:teuthology.orchestra.run.smithi083.stdout:Total 800 kB/s | 176 MB 03:44 2024-03-25T18:25:06.791 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T18:25:06.794 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T18:25:06.795 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T18:25:07.850 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T18:25:07.850 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T18:25:26.060 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T18:25:26.244 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:25:31.097 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:25:31.368 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:25:31.369 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T18:25:31.369 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T18:25:31.369 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:25:31.369 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T18:25:31.369 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T18:25:31.546 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-test-debuginfo 2024-03-25T18:25:32.103 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:35:40 ago on Mon 25 Mar 2024 05:49:52 PM UTC. 2024-03-25T18:25:32.250 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T18:25:32.251 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T18:25:32.251 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-25T18:25:32.251 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T18:25:32.251 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T18:25:32.251 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T18:25:32.251 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T18:25:32.251 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T18:25:32.251 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T18:25:32.252 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T18:25:32.252 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T18:25:32.252 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 672 M 2024-03-25T18:25:32.252 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 3.1 G 2024-03-25T18:25:32.252 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T18:25:40.113 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-25T18:25:40.223 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:25:45.402 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:25:45.655 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:25:45.655 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:25:45.655 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-25T18:25:45.655 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:25:45.655 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:25:45.655 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-25T18:25:45.920 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T18:25:46.485 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:31:52 ago on Mon 25 Mar 2024 05:53:54 PM UTC. 2024-03-25T18:25:46.634 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-25T18:25:46.635 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================== 2024-03-25T18:25:46.635 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-03-25T18:25:46.635 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================== 2024-03-25T18:25:46.635 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-25T18:25:46.635 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T18:25:46.635 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:25:46.636 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-25T18:25:46.636 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================== 2024-03-25T18:25:46.636 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-03-25T18:25:46.636 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:25:46.636 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 1.4 M 2024-03-25T18:25:46.636 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 6.4 M 2024-03-25T18:25:46.636 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-25T18:25:49.960 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 440 kB/s | 1.4 MB 00:03 2024-03-25T18:25:49.960 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:25:49.961 INFO:teuthology.orchestra.run.smithi103.stdout:Total 439 kB/s | 1.4 MB 00:03 2024-03-25T18:25:49.961 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-25T18:25:49.965 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-25T18:25:49.965 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-25T18:25:50.010 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-25T18:25:50.011 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-25T18:25:50.288 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-25T18:25:50.392 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:25:50.793 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:25:51.062 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:25:51.063 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:25:51.063 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-25T18:25:51.063 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:25:51.063 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:25:51.063 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-25T18:25:51.254 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T18:25:51.809 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:31:57 ago on Mon 25 Mar 2024 05:53:54 PM UTC. 2024-03-25T18:25:51.955 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-25T18:25:51.956 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T18:25:51.956 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-03-25T18:25:51.956 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T18:25:51.956 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-25T18:25:51.956 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T18:25:51.956 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:25:51.956 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-25T18:25:51.956 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T18:25:51.956 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-03-25T18:25:51.957 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:25:51.957 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 176 M 2024-03-25T18:25:51.957 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 753 M 2024-03-25T18:25:51.957 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-25T18:29:18.918 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 869 kB/s | 176 MB 03:26 2024-03-25T18:29:18.919 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:29:18.919 INFO:teuthology.orchestra.run.smithi103.stdout:Total 869 kB/s | 176 MB 03:26 2024-03-25T18:29:18.919 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-25T18:29:18.921 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-25T18:29:18.921 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-25T18:29:19.979 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-25T18:29:19.979 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-25T18:29:38.243 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-25T18:29:38.347 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:29:43.167 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:29:43.435 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:29:43.436 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:29:43.436 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-25T18:29:43.436 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:29:43.436 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:29:43.436 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-25T18:29:43.655 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-test-debuginfo 2024-03-25T18:29:44.220 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:35:50 ago on Mon 25 Mar 2024 05:53:54 PM UTC. 2024-03-25T18:29:44.369 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-25T18:29:44.370 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T18:29:44.370 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-03-25T18:29:44.370 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T18:29:44.370 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-25T18:29:44.370 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T18:29:44.370 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:29:44.371 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-25T18:29:44.371 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T18:29:44.371 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-03-25T18:29:44.371 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:29:44.371 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 672 M 2024-03-25T18:29:44.371 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 3.1 G 2024-03-25T18:29:44.371 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-25T18:41:52.255 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 702 kB/s | 672 MB 16:20 2024-03-25T18:41:52.256 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:41:52.256 INFO:teuthology.orchestra.run.smithi083.stdout:Total 702 kB/s | 672 MB 16:20 2024-03-25T18:41:52.256 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T18:41:52.260 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T18:41:52.260 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T18:41:56.220 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T18:41:56.220 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T18:43:07.007 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T18:43:07.190 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:43:16.259 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:43:16.521 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:43:16.521 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T18:43:16.521 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T18:43:16.521 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:43:16.521 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T18:43:16.521 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T18:43:16.797 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-base-debuginfo 2024-03-25T18:43:17.358 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:53:25 ago on Mon 25 Mar 2024 05:49:52 PM UTC. 2024-03-25T18:43:17.454 INFO:teuthology.orchestra.run.smithi083.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:43:17.507 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T18:43:17.508 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-03-25T18:43:17.508 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T18:43:17.576 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T18:43:18.135 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:53:26 ago on Mon 25 Mar 2024 05:49:52 PM UTC. 2024-03-25T18:43:18.284 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T18:43:18.285 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T18:43:18.285 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-25T18:43:18.285 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T18:43:18.286 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T18:43:18.286 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T18:43:18.286 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T18:43:18.286 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T18:43:18.286 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T18:43:18.286 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T18:43:18.286 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T18:43:18.286 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 20 M 2024-03-25T18:43:18.286 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 103 M 2024-03-25T18:43:18.286 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T18:43:58.823 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 498 kB/s | 20 MB 00:40 2024-03-25T18:43:58.824 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:43:58.824 INFO:teuthology.orchestra.run.smithi083.stdout:Total 498 kB/s | 20 MB 00:40 2024-03-25T18:43:58.824 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T18:43:58.828 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T18:43:58.828 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T18:43:58.977 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T18:43:58.977 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T18:44:01.193 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T18:44:01.345 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:44:02.420 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:44:02.672 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:44:02.673 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T18:44:02.673 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T18:44:02.673 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:44:02.673 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T18:44:02.673 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T18:44:02.839 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T18:44:03.398 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:54:11 ago on Mon 25 Mar 2024 05:49:52 PM UTC. 2024-03-25T18:44:03.547 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T18:44:03.548 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T18:44:03.548 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-25T18:44:03.548 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T18:44:03.548 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T18:44:03.548 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T18:44:03.549 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T18:44:03.549 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T18:44:03.549 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T18:44:03.549 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T18:44:03.549 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T18:44:03.549 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 48 M 2024-03-25T18:44:03.549 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 173 M 2024-03-25T18:44:03.549 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T18:45:53.506 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 444 kB/s | 48 MB 01:49 2024-03-25T18:45:53.506 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:45:53.506 INFO:teuthology.orchestra.run.smithi083.stdout:Total 444 kB/s | 48 MB 01:49 2024-03-25T18:45:53.507 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T18:45:53.509 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T18:45:53.509 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T18:45:53.822 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T18:45:53.823 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T18:45:58.547 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T18:45:58.678 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:46:00.033 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:46:00.315 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:46:00.315 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T18:46:00.315 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T18:46:00.315 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:46:00.315 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T18:46:00.315 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T18:46:00.486 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T18:46:01.048 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:56:09 ago on Mon 25 Mar 2024 05:49:52 PM UTC. 2024-03-25T18:46:01.196 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T18:46:01.196 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T18:46:01.197 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-25T18:46:01.197 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T18:46:01.197 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T18:46:01.197 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T18:46:01.197 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T18:46:01.197 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T18:46:01.197 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T18:46:01.197 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T18:46:01.197 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T18:46:01.197 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 99 M 2024-03-25T18:46:01.197 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 449 M 2024-03-25T18:46:01.197 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T18:49:07.929 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 540 kB/s | 99 MB 03:06 2024-03-25T18:49:07.929 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:49:07.930 INFO:teuthology.orchestra.run.smithi083.stdout:Total 540 kB/s | 99 MB 03:06 2024-03-25T18:49:07.930 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T18:49:07.932 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T18:49:07.933 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T18:49:08.542 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T18:49:08.543 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T18:49:18.816 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T18:49:18.917 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:49:21.873 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:49:22.146 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:49:22.146 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T18:49:22.146 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T18:49:22.147 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:49:22.147 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T18:49:22.147 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T18:49:22.351 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T18:49:22.914 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:59:30 ago on Mon 25 Mar 2024 05:49:52 PM UTC. 2024-03-25T18:49:23.063 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T18:49:23.063 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T18:49:23.063 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-25T18:49:23.064 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T18:49:23.064 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T18:49:23.064 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T18:49:23.064 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T18:49:23.064 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T18:49:23.064 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T18:49:23.064 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T18:49:23.064 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T18:49:23.065 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 287 M 2024-03-25T18:49:23.065 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 1.2 G 2024-03-25T18:49:23.065 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T18:54:00.828 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 473 kB/s | 672 MB 24:16 2024-03-25T18:54:00.829 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:54:00.829 INFO:teuthology.orchestra.run.smithi103.stdout:Total 473 kB/s | 672 MB 24:16 2024-03-25T18:54:00.829 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-25T18:54:00.833 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-25T18:54:00.833 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-25T18:54:04.807 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-25T18:54:04.807 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-25T18:55:15.877 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-25T18:55:16.081 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:55:26.058 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:55:26.321 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:55:26.321 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:55:26.321 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-25T18:55:26.321 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:55:26.321 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:55:26.321 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-25T18:55:26.651 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-base-debuginfo 2024-03-25T18:55:27.215 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 1:01:33 ago on Mon 25 Mar 2024 05:53:54 PM UTC. 2024-03-25T18:55:27.311 INFO:teuthology.orchestra.run.smithi103.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:55:27.363 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-25T18:55:27.364 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-03-25T18:55:27.364 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-25T18:55:27.445 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T18:55:28.009 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 1:01:34 ago on Mon 25 Mar 2024 05:53:54 PM UTC. 2024-03-25T18:55:28.158 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-25T18:55:28.159 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T18:55:28.159 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-03-25T18:55:28.159 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T18:55:28.160 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-25T18:55:28.160 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T18:55:28.160 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:55:28.160 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-25T18:55:28.160 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T18:55:28.160 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-03-25T18:55:28.160 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:55:28.160 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 20 M 2024-03-25T18:55:28.160 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 103 M 2024-03-25T18:55:28.160 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-25T18:56:12.249 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 458 kB/s | 20 MB 00:44 2024-03-25T18:56:12.249 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:56:12.249 INFO:teuthology.orchestra.run.smithi103.stdout:Total 458 kB/s | 20 MB 00:44 2024-03-25T18:56:12.249 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-25T18:56:12.252 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-25T18:56:12.252 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-25T18:56:12.402 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-25T18:56:12.402 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-25T18:56:14.657 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-25T18:56:14.837 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:56:15.845 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:56:16.107 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:56:16.108 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:56:16.108 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-25T18:56:16.108 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:56:16.108 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:56:16.108 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-25T18:56:16.292 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T18:56:16.856 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 1:02:22 ago on Mon 25 Mar 2024 05:53:54 PM UTC. 2024-03-25T18:56:17.006 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-25T18:56:17.007 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T18:56:17.007 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-03-25T18:56:17.007 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T18:56:17.007 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-25T18:56:17.007 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T18:56:17.007 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:56:17.008 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-25T18:56:17.008 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T18:56:17.008 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-03-25T18:56:17.008 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:56:17.008 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 48 M 2024-03-25T18:56:17.008 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 173 M 2024-03-25T18:56:17.008 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-25T18:58:07.296 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 560 kB/s | 287 MB 08:44 2024-03-25T18:58:07.296 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:58:07.296 INFO:teuthology.orchestra.run.smithi083.stdout:Total 560 kB/s | 287 MB 08:44 2024-03-25T18:58:07.297 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T18:58:07.300 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T18:58:07.300 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T18:58:09.003 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T18:58:09.003 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T18:58:28.636 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 371 kB/s | 48 MB 02:11 2024-03-25T18:58:28.637 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:58:28.638 INFO:teuthology.orchestra.run.smithi103.stdout:Total 371 kB/s | 48 MB 02:11 2024-03-25T18:58:28.638 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-25T18:58:28.641 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-25T18:58:28.641 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-25T18:58:28.953 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-25T18:58:28.953 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-25T18:58:33.697 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-25T18:58:33.845 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:58:35.313 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:58:35.573 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:58:35.573 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:58:35.573 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-25T18:58:35.573 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:58:35.573 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:58:35.573 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-25T18:58:35.769 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T18:58:36.328 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 1:04:42 ago on Mon 25 Mar 2024 05:53:54 PM UTC. 2024-03-25T18:58:36.475 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-25T18:58:36.476 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T18:58:36.476 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-03-25T18:58:36.476 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T18:58:36.476 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-25T18:58:36.476 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T18:58:36.476 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:58:36.476 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-25T18:58:36.476 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T18:58:36.476 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-03-25T18:58:36.476 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T18:58:36.477 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 99 M 2024-03-25T18:58:36.477 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 449 M 2024-03-25T18:58:36.477 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-25T18:58:38.259 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T18:58:38.357 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:58:46.025 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:58:46.302 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:58:46.303 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T18:58:46.303 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T18:58:46.303 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:58:46.303 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T18:58:46.303 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T18:58:46.495 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T18:58:47.054 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 1:08:55 ago on Mon 25 Mar 2024 05:49:52 PM UTC. 2024-03-25T18:58:47.201 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T18:58:47.202 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T18:58:47.202 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-25T18:58:47.202 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T18:58:47.202 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T18:58:47.203 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T18:58:47.203 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T18:58:47.203 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T18:58:47.203 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T18:58:47.203 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T18:58:47.203 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T18:58:47.203 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 5.9 M 2024-03-25T18:58:47.203 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 28 M 2024-03-25T18:58:47.203 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T18:59:05.593 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 331 kB/s | 5.9 MB 00:18 2024-03-25T18:59:05.594 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:59:05.594 INFO:teuthology.orchestra.run.smithi083.stdout:Total 331 kB/s | 5.9 MB 00:18 2024-03-25T18:59:05.595 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T18:59:05.598 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T18:59:05.598 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T18:59:05.670 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T18:59:05.671 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T18:59:06.398 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T18:59:06.509 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:59:07.042 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:59:07.316 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:59:07.316 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T18:59:07.316 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T18:59:07.316 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:59:07.317 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T18:59:07.317 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T18:59:07.511 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install librados-devel-debuginfo 2024-03-25T18:59:08.069 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 1:09:16 ago on Mon 25 Mar 2024 05:49:52 PM UTC. 2024-03-25T18:59:08.220 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T18:59:08.221 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T18:59:08.221 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-25T18:59:08.221 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T18:59:08.221 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T18:59:08.221 INFO:teuthology.orchestra.run.smithi083.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T18:59:08.221 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T18:59:08.221 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T18:59:08.221 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T18:59:08.221 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T18:59:08.221 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T18:59:08.221 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 109 k 2024-03-25T18:59:08.221 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 440 k 2024-03-25T18:59:08.222 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T18:59:08.996 INFO:teuthology.orchestra.run.smithi083.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 140 kB/s | 109 kB 00:00 2024-03-25T18:59:08.997 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:59:08.997 INFO:teuthology.orchestra.run.smithi083.stdout:Total 140 kB/s | 109 kB 00:00 2024-03-25T18:59:08.997 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T18:59:09.001 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T18:59:09.001 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T18:59:09.037 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T18:59:09.037 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T18:59:09.153 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T18:59:09.262 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:59:09.639 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:59:09.889 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:59:09.889 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T18:59:09.889 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T18:59:09.889 INFO:teuthology.orchestra.run.smithi083.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:59:09.889 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T18:59:09.889 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T18:59:10.065 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T18:59:10.621 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 1:09:18 ago on Mon 25 Mar 2024 05:49:52 PM UTC. 2024-03-25T18:59:10.769 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T18:59:10.770 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T18:59:10.770 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-25T18:59:10.770 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T18:59:10.770 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T18:59:10.770 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T18:59:10.770 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T18:59:10.770 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T18:59:10.770 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T18:59:10.771 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T18:59:10.771 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T18:59:10.771 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 5.4 M 2024-03-25T18:59:10.771 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 25 M 2024-03-25T18:59:10.771 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T18:59:30.339 INFO:teuthology.orchestra.run.smithi083.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 280 kB/s | 5.4 MB 00:19 2024-03-25T18:59:30.340 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:59:30.340 INFO:teuthology.orchestra.run.smithi083.stdout:Total 280 kB/s | 5.4 MB 00:19 2024-03-25T18:59:30.340 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T18:59:30.344 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T18:59:30.345 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T18:59:30.413 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T18:59:30.414 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T18:59:31.078 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T18:59:31.187 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:59:31.707 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:59:31.978 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:59:31.978 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T18:59:31.978 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T18:59:31.978 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:59:31.978 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T18:59:31.978 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T18:59:32.162 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install librados2-debuginfo 2024-03-25T18:59:32.725 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 1:09:40 ago on Mon 25 Mar 2024 05:49:52 PM UTC. 2024-03-25T18:59:32.874 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T18:59:32.875 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T18:59:32.875 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-25T18:59:32.875 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T18:59:32.875 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T18:59:32.875 INFO:teuthology.orchestra.run.smithi083.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T18:59:32.875 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T18:59:32.876 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T18:59:32.876 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T18:59:32.876 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T18:59:32.876 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T18:59:32.876 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 44 M 2024-03-25T18:59:32.876 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 253 M 2024-03-25T18:59:32.876 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T19:01:41.204 INFO:teuthology.orchestra.run.smithi083.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 353 kB/s | 44 MB 02:08 2024-03-25T19:01:41.205 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:01:41.205 INFO:teuthology.orchestra.run.smithi083.stdout:Total 353 kB/s | 44 MB 02:08 2024-03-25T19:01:41.205 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T19:01:41.208 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T19:01:41.208 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T19:01:41.507 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T19:01:41.508 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T19:01:46.551 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T19:01:46.653 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:01:48.568 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:01:48.830 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:01:48.830 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:01:48.830 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T19:01:48.830 INFO:teuthology.orchestra.run.smithi083.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:01:48.831 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:01:48.831 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T19:01:49.026 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install librbd1-debuginfo 2024-03-25T19:01:49.591 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 1:11:57 ago on Mon 25 Mar 2024 05:49:52 PM UTC. 2024-03-25T19:01:49.739 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T19:01:49.740 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T19:01:49.741 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-25T19:01:49.741 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T19:01:49.741 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T19:01:49.741 INFO:teuthology.orchestra.run.smithi083.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:01:49.741 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:01:49.741 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T19:01:49.741 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T19:01:49.741 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T19:01:49.741 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:01:49.741 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 52 M 2024-03-25T19:01:49.741 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 308 M 2024-03-25T19:01:49.741 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T19:02:15.292 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 461 kB/s | 99 MB 03:38 2024-03-25T19:02:15.293 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:02:15.293 INFO:teuthology.orchestra.run.smithi103.stdout:Total 461 kB/s | 99 MB 03:38 2024-03-25T19:02:15.293 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-25T19:02:15.296 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-25T19:02:15.296 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-25T19:02:15.906 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-25T19:02:15.907 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-25T19:02:26.129 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-25T19:02:26.230 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:02:29.259 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:02:29.527 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:02:29.527 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:02:29.527 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-25T19:02:29.527 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:02:29.527 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:02:29.527 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-25T19:02:29.725 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T19:02:30.286 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 1:08:36 ago on Mon 25 Mar 2024 05:53:54 PM UTC. 2024-03-25T19:02:30.433 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-25T19:02:30.434 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T19:02:30.434 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-03-25T19:02:30.434 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T19:02:30.434 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-25T19:02:30.434 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T19:02:30.434 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:02:30.434 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-25T19:02:30.434 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T19:02:30.434 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-03-25T19:02:30.434 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:02:30.434 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 287 M 2024-03-25T19:02:30.434 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 1.2 G 2024-03-25T19:02:30.435 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-25T19:03:47.594 INFO:teuthology.orchestra.run.smithi083.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 450 kB/s | 52 MB 01:57 2024-03-25T19:03:47.594 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:03:47.595 INFO:teuthology.orchestra.run.smithi083.stdout:Total 450 kB/s | 52 MB 01:57 2024-03-25T19:03:47.595 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T19:03:47.598 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T19:03:47.598 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T19:03:47.941 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T19:03:47.941 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T19:03:53.923 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T19:03:54.300 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:03:56.467 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:03:56.727 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:03:56.727 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:03:56.727 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T19:03:56.727 INFO:teuthology.orchestra.run.smithi083.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:03:56.727 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:03:56.727 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T19:03:56.917 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T19:03:57.476 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 1:14:05 ago on Mon 25 Mar 2024 05:49:52 PM UTC. 2024-03-25T19:03:57.627 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T19:03:57.628 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T19:03:57.628 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-25T19:03:57.628 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T19:03:57.629 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T19:03:57.629 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T19:03:57.629 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:03:57.629 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T19:03:57.629 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T19:03:57.629 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T19:03:57.629 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:03:57.629 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 615 k 2024-03-25T19:03:57.629 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 2.1 M 2024-03-25T19:03:57.629 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T19:03:59.449 INFO:teuthology.orchestra.run.smithi083.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 338 kB/s | 615 kB 00:01 2024-03-25T19:03:59.449 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:03:59.450 INFO:teuthology.orchestra.run.smithi083.stdout:Total 338 kB/s | 615 kB 00:01 2024-03-25T19:03:59.450 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T19:03:59.454 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T19:03:59.454 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T19:03:59.495 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T19:03:59.495 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T19:03:59.665 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T19:03:59.761 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:04:00.128 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:04:00.392 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:04:00.393 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:04:00.393 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T19:04:00.393 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:04:00.393 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:04:00.393 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T19:04:00.571 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install python3-rados-debuginfo 2024-03-25T19:04:01.130 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 1:14:09 ago on Mon 25 Mar 2024 05:49:52 PM UTC. 2024-03-25T19:04:01.282 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T19:04:01.283 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T19:04:01.283 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-25T19:04:01.283 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T19:04:01.283 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T19:04:01.283 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T19:04:01.283 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:04:01.283 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T19:04:01.283 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T19:04:01.284 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T19:04:01.284 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:04:01.284 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 771 k 2024-03-25T19:04:01.284 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 2.7 M 2024-03-25T19:04:01.284 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T19:04:02.874 INFO:teuthology.orchestra.run.smithi083.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 485 kB/s | 771 kB 00:01 2024-03-25T19:04:02.874 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:04:02.875 INFO:teuthology.orchestra.run.smithi083.stdout:Total 485 kB/s | 771 kB 00:01 2024-03-25T19:04:02.875 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T19:04:02.879 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T19:04:02.879 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T19:04:02.921 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T19:04:02.922 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T19:04:03.123 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T19:04:03.239 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:04:03.615 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:04:03.901 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:04:03.901 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:04:03.901 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T19:04:03.901 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:04:03.901 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:04:03.901 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T19:04:04.102 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T19:04:04.658 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 1:14:12 ago on Mon 25 Mar 2024 05:49:52 PM UTC. 2024-03-25T19:04:04.805 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T19:04:04.806 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T19:04:04.806 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-25T19:04:04.806 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T19:04:04.806 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T19:04:04.806 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T19:04:04.806 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:04:04.807 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T19:04:04.807 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T19:04:04.807 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T19:04:04.807 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:04:04.807 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 1.1 M 2024-03-25T19:04:04.807 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 4.3 M 2024-03-25T19:04:04.807 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T19:04:06.737 INFO:teuthology.orchestra.run.smithi083.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 596 kB/s | 1.1 MB 00:01 2024-03-25T19:04:06.738 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:04:06.738 INFO:teuthology.orchestra.run.smithi083.stdout:Total 596 kB/s | 1.1 MB 00:01 2024-03-25T19:04:06.738 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T19:04:06.742 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T19:04:06.742 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T19:04:06.786 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T19:04:06.786 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T19:04:07.009 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T19:04:07.104 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:04:07.511 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:04:07.774 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:04:07.774 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:04:07.774 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T19:04:07.774 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:04:07.774 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:04:07.774 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T19:04:07.980 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T19:04:08.536 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 1:14:16 ago on Mon 25 Mar 2024 05:49:52 PM UTC. 2024-03-25T19:04:08.683 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T19:04:08.684 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T19:04:08.684 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-25T19:04:08.684 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T19:04:08.684 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T19:04:08.684 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T19:04:08.684 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:04:08.684 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T19:04:08.684 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T19:04:08.684 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T19:04:08.684 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:04:08.685 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 337 k 2024-03-25T19:04:08.685 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 1.0 M 2024-03-25T19:04:08.685 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T19:04:10.349 INFO:teuthology.orchestra.run.smithi083.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 203 kB/s | 337 kB 00:01 2024-03-25T19:04:10.350 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:04:10.350 INFO:teuthology.orchestra.run.smithi083.stdout:Total 203 kB/s | 337 kB 00:01 2024-03-25T19:04:10.350 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T19:04:10.354 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T19:04:10.354 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T19:04:10.394 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T19:04:10.394 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T19:04:10.536 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T19:04:10.639 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:04:11.022 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:04:11.274 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:04:11.274 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:04:11.274 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T19:04:11.274 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:04:11.274 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:04:11.275 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T19:04:11.488 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T19:04:12.046 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 1:14:20 ago on Mon 25 Mar 2024 05:49:52 PM UTC. 2024-03-25T19:04:12.194 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T19:04:12.194 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T19:04:12.195 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-25T19:04:12.195 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T19:04:12.195 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T19:04:12.195 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T19:04:12.195 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:04:12.195 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T19:04:12.195 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T19:04:12.195 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T19:04:12.195 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:04:12.195 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 509 k 2024-03-25T19:04:12.195 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 2.5 M 2024-03-25T19:04:12.195 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T19:04:14.169 INFO:teuthology.orchestra.run.smithi083.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 258 kB/s | 509 kB 00:01 2024-03-25T19:04:14.169 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:04:14.170 INFO:teuthology.orchestra.run.smithi083.stdout:Total 258 kB/s | 509 kB 00:01 2024-03-25T19:04:14.170 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T19:04:14.174 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T19:04:14.174 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T19:04:14.215 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T19:04:14.215 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T19:04:14.374 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T19:04:14.475 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:04:14.857 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:04:15.116 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:04:15.116 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:04:15.116 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T19:04:15.117 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:04:15.117 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:04:15.117 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T19:04:15.310 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T19:04:15.870 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 1:14:23 ago on Mon 25 Mar 2024 05:49:52 PM UTC. 2024-03-25T19:04:16.018 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T19:04:16.018 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T19:04:16.018 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-25T19:04:16.018 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T19:04:16.018 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T19:04:16.019 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:04:16.019 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:04:16.019 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T19:04:16.019 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T19:04:16.019 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T19:04:16.019 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:04:16.019 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 52 M 2024-03-25T19:04:16.020 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 298 M 2024-03-25T19:04:16.020 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T19:05:59.536 INFO:teuthology.orchestra.run.smithi083.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 510 kB/s | 52 MB 01:43 2024-03-25T19:05:59.536 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:05:59.537 INFO:teuthology.orchestra.run.smithi083.stdout:Total 510 kB/s | 52 MB 01:43 2024-03-25T19:05:59.537 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T19:05:59.540 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T19:05:59.540 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T19:05:59.875 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T19:05:59.875 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T19:06:05.759 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T19:06:05.982 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:06:08.058 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:06:08.331 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:06:08.331 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:06:08.331 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T19:06:08.331 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:06:08.331 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:06:08.332 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T19:06:08.525 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T19:06:09.085 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 1:16:17 ago on Mon 25 Mar 2024 05:49:52 PM UTC. 2024-03-25T19:06:09.235 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T19:06:09.236 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T19:06:09.236 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-25T19:06:09.236 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T19:06:09.236 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T19:06:09.236 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T19:06:09.236 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:06:09.236 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T19:06:09.236 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T19:06:09.236 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T19:06:09.236 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:06:09.237 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 1.3 M 2024-03-25T19:06:09.237 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 5.5 M 2024-03-25T19:06:09.237 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T19:06:14.853 INFO:teuthology.orchestra.run.smithi083.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 228 kB/s | 1.3 MB 00:05 2024-03-25T19:06:14.854 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:06:14.855 INFO:teuthology.orchestra.run.smithi083.stdout:Total 228 kB/s | 1.3 MB 00:05 2024-03-25T19:06:14.855 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T19:06:14.859 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T19:06:14.859 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T19:06:14.904 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T19:06:14.904 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T19:06:15.144 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T19:06:15.243 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:06:15.663 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:06:15.924 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:06:15.924 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:06:15.925 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T19:06:15.925 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:06:15.925 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:06:15.925 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T19:06:16.098 DEBUG:teuthology.parallel:result is None 2024-03-25T19:12:03.927 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 512 kB/s | 287 MB 09:33 2024-03-25T19:12:03.927 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:12:03.927 INFO:teuthology.orchestra.run.smithi103.stdout:Total 512 kB/s | 287 MB 09:33 2024-03-25T19:12:03.927 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-25T19:12:03.930 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-25T19:12:03.930 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-25T19:12:05.645 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-25T19:12:05.645 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-25T19:12:35.075 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-25T19:12:35.180 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:12:43.040 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:12:43.326 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:12:43.326 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:12:43.326 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-25T19:12:43.326 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:12:43.327 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:12:43.327 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-25T19:12:43.537 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T19:12:44.097 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 1:18:50 ago on Mon 25 Mar 2024 05:53:54 PM UTC. 2024-03-25T19:12:44.245 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-25T19:12:44.246 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T19:12:44.246 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-03-25T19:12:44.246 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T19:12:44.246 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-25T19:12:44.246 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T19:12:44.246 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:12:44.247 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-25T19:12:44.247 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T19:12:44.247 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-03-25T19:12:44.247 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:12:44.247 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 5.9 M 2024-03-25T19:12:44.247 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 28 M 2024-03-25T19:12:44.247 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-25T19:12:53.817 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 636 kB/s | 5.9 MB 00:09 2024-03-25T19:12:53.818 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:12:53.818 INFO:teuthology.orchestra.run.smithi103.stdout:Total 636 kB/s | 5.9 MB 00:09 2024-03-25T19:12:53.818 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-25T19:12:53.822 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-25T19:12:53.822 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-25T19:12:53.894 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-25T19:12:53.894 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-25T19:12:54.637 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-25T19:12:54.782 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:12:55.423 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:12:55.694 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:12:55.694 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:12:55.694 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-25T19:12:55.694 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:12:55.694 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:12:55.694 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-25T19:12:55.892 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install librados-devel-debuginfo 2024-03-25T19:12:56.453 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 1:19:02 ago on Mon 25 Mar 2024 05:53:54 PM UTC. 2024-03-25T19:12:56.606 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-25T19:12:56.607 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T19:12:56.607 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-03-25T19:12:56.607 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T19:12:56.607 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-25T19:12:56.607 INFO:teuthology.orchestra.run.smithi103.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T19:12:56.608 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:12:56.608 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-25T19:12:56.608 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T19:12:56.608 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-03-25T19:12:56.608 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:12:56.608 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 109 k 2024-03-25T19:12:56.608 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 440 k 2024-03-25T19:12:56.608 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-25T19:12:57.229 INFO:teuthology.orchestra.run.smithi103.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 175 kB/s | 109 kB 00:00 2024-03-25T19:12:57.229 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:12:57.229 INFO:teuthology.orchestra.run.smithi103.stdout:Total 175 kB/s | 109 kB 00:00 2024-03-25T19:12:57.229 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-25T19:12:57.232 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-25T19:12:57.232 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-25T19:12:57.268 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-25T19:12:57.269 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-25T19:12:57.388 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-25T19:12:57.502 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:12:57.894 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:12:58.152 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:12:58.153 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:12:58.153 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-25T19:12:58.153 INFO:teuthology.orchestra.run.smithi103.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:12:58.153 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:12:58.153 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-25T19:12:58.352 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T19:12:58.911 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 1:19:04 ago on Mon 25 Mar 2024 05:53:54 PM UTC. 2024-03-25T19:12:59.059 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-25T19:12:59.060 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T19:12:59.060 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-03-25T19:12:59.060 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T19:12:59.060 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-25T19:12:59.060 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T19:12:59.060 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:12:59.060 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-25T19:12:59.060 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T19:12:59.060 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-03-25T19:12:59.060 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:12:59.061 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 5.4 M 2024-03-25T19:12:59.061 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 25 M 2024-03-25T19:12:59.061 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-25T19:13:13.555 INFO:teuthology.orchestra.run.smithi103.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 378 kB/s | 5.4 MB 00:14 2024-03-25T19:13:13.556 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:13:13.556 INFO:teuthology.orchestra.run.smithi103.stdout:Total 378 kB/s | 5.4 MB 00:14 2024-03-25T19:13:13.556 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-25T19:13:13.560 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-25T19:13:13.560 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-25T19:13:13.629 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-25T19:13:13.630 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-25T19:13:14.301 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-25T19:13:14.446 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:13:15.028 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:13:15.293 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:13:15.294 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:13:15.294 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-25T19:13:15.294 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:13:15.294 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:13:15.294 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-25T19:13:15.494 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install librados2-debuginfo 2024-03-25T19:13:16.057 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 1:19:22 ago on Mon 25 Mar 2024 05:53:54 PM UTC. 2024-03-25T19:13:16.205 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-25T19:13:16.206 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T19:13:16.206 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-03-25T19:13:16.206 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T19:13:16.207 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-25T19:13:16.207 INFO:teuthology.orchestra.run.smithi103.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T19:13:16.207 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:13:16.207 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-25T19:13:16.207 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T19:13:16.207 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-03-25T19:13:16.207 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:13:16.207 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 44 M 2024-03-25T19:13:16.207 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 253 M 2024-03-25T19:13:16.207 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-25T19:14:57.957 INFO:teuthology.orchestra.run.smithi103.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 446 kB/s | 44 MB 01:41 2024-03-25T19:14:57.957 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:14:57.957 INFO:teuthology.orchestra.run.smithi103.stdout:Total 446 kB/s | 44 MB 01:41 2024-03-25T19:14:57.957 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-25T19:14:57.960 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-25T19:14:57.960 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-25T19:14:58.261 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-25T19:14:58.262 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-25T19:15:03.319 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-25T19:15:03.421 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:15:05.339 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:15:05.618 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:15:05.618 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:15:05.618 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-25T19:15:05.618 INFO:teuthology.orchestra.run.smithi103.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:15:05.618 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:15:05.618 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-25T19:15:05.836 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install librbd1-debuginfo 2024-03-25T19:15:06.402 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 1:21:12 ago on Mon 25 Mar 2024 05:53:54 PM UTC. 2024-03-25T19:15:06.551 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-25T19:15:06.552 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T19:15:06.552 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-03-25T19:15:06.552 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T19:15:06.552 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-25T19:15:06.552 INFO:teuthology.orchestra.run.smithi103.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:15:06.552 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:15:06.552 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-25T19:15:06.552 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T19:15:06.552 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-03-25T19:15:06.552 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:15:06.553 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 52 M 2024-03-25T19:15:06.553 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 308 M 2024-03-25T19:15:06.553 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-25T19:16:34.533 INFO:teuthology.orchestra.run.smithi103.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 603 kB/s | 52 MB 01:27 2024-03-25T19:16:34.534 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:16:34.534 INFO:teuthology.orchestra.run.smithi103.stdout:Total 603 kB/s | 52 MB 01:27 2024-03-25T19:16:34.534 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-25T19:16:34.537 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-25T19:16:34.537 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-25T19:16:34.883 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-25T19:16:34.883 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-25T19:16:40.897 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-25T19:16:41.106 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:16:43.456 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:16:43.730 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:16:43.730 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:16:43.730 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-25T19:16:43.730 INFO:teuthology.orchestra.run.smithi103.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:16:43.730 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:16:43.730 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-25T19:16:43.938 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T19:16:44.498 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 1:22:50 ago on Mon 25 Mar 2024 05:53:54 PM UTC. 2024-03-25T19:16:44.647 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-25T19:16:44.648 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T19:16:44.648 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-03-25T19:16:44.648 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T19:16:44.649 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-25T19:16:44.649 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T19:16:44.649 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:16:44.649 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-25T19:16:44.649 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T19:16:44.649 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-03-25T19:16:44.649 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:16:44.649 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 615 k 2024-03-25T19:16:44.649 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 2.1 M 2024-03-25T19:16:44.649 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-25T19:16:45.818 INFO:teuthology.orchestra.run.smithi103.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 526 kB/s | 615 kB 00:01 2024-03-25T19:16:45.819 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:16:45.819 INFO:teuthology.orchestra.run.smithi103.stdout:Total 525 kB/s | 615 kB 00:01 2024-03-25T19:16:45.820 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-25T19:16:45.823 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-25T19:16:45.823 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-25T19:16:45.863 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-25T19:16:45.863 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-25T19:16:46.028 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-25T19:16:46.126 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:16:46.526 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:16:46.796 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:16:46.796 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:16:46.796 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-25T19:16:46.796 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:16:46.796 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:16:46.796 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-25T19:16:46.991 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install python3-rados-debuginfo 2024-03-25T19:16:47.556 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 1:22:53 ago on Mon 25 Mar 2024 05:53:54 PM UTC. 2024-03-25T19:16:47.708 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-25T19:16:47.708 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T19:16:47.709 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-03-25T19:16:47.709 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T19:16:47.709 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-25T19:16:47.709 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T19:16:47.709 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:16:47.709 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-25T19:16:47.709 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T19:16:47.709 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-03-25T19:16:47.709 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:16:47.709 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 771 k 2024-03-25T19:16:47.709 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 2.7 M 2024-03-25T19:16:47.710 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-25T19:16:50.130 INFO:teuthology.orchestra.run.smithi103.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 319 kB/s | 771 kB 00:02 2024-03-25T19:16:50.130 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:16:50.130 INFO:teuthology.orchestra.run.smithi103.stdout:Total 318 kB/s | 771 kB 00:02 2024-03-25T19:16:50.131 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-25T19:16:50.135 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-25T19:16:50.135 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-25T19:16:50.178 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-25T19:16:50.178 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-25T19:16:50.362 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-25T19:16:50.471 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:16:50.880 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:16:51.146 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:16:51.147 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:16:51.147 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-25T19:16:51.147 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:16:51.147 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:16:51.147 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-25T19:16:51.395 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T19:16:51.959 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 1:22:57 ago on Mon 25 Mar 2024 05:53:54 PM UTC. 2024-03-25T19:16:52.107 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-25T19:16:52.108 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T19:16:52.108 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-03-25T19:16:52.108 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T19:16:52.108 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-25T19:16:52.108 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T19:16:52.108 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:16:52.108 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-25T19:16:52.108 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T19:16:52.108 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-03-25T19:16:52.108 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:16:52.109 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 1.1 M 2024-03-25T19:16:52.109 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 4.3 M 2024-03-25T19:16:52.109 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-25T19:16:55.057 INFO:teuthology.orchestra.run.smithi103.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 390 kB/s | 1.1 MB 00:02 2024-03-25T19:16:55.058 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:16:55.059 INFO:teuthology.orchestra.run.smithi103.stdout:Total 390 kB/s | 1.1 MB 00:02 2024-03-25T19:16:55.059 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-25T19:16:55.063 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-25T19:16:55.063 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-25T19:16:55.107 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-25T19:16:55.108 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-25T19:16:55.331 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-25T19:16:55.440 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:16:55.855 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:16:56.129 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:16:56.129 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:16:56.129 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-25T19:16:56.129 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:16:56.129 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:16:56.129 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-25T19:16:56.363 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T19:16:56.926 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 1:23:02 ago on Mon 25 Mar 2024 05:53:54 PM UTC. 2024-03-25T19:16:57.074 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-25T19:16:57.075 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T19:16:57.075 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-03-25T19:16:57.075 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T19:16:57.075 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-25T19:16:57.075 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T19:16:57.075 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:16:57.075 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-25T19:16:57.075 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T19:16:57.075 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-03-25T19:16:57.076 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:16:57.076 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 337 k 2024-03-25T19:16:57.076 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 1.0 M 2024-03-25T19:16:57.076 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-25T19:16:58.811 INFO:teuthology.orchestra.run.smithi103.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 194 kB/s | 337 kB 00:01 2024-03-25T19:16:58.811 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:16:58.812 INFO:teuthology.orchestra.run.smithi103.stdout:Total 194 kB/s | 337 kB 00:01 2024-03-25T19:16:58.812 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-25T19:16:58.816 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-25T19:16:58.816 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-25T19:16:58.857 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-25T19:16:58.857 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-25T19:16:59.001 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-25T19:16:59.093 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:16:59.497 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:16:59.770 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:16:59.771 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:16:59.771 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-25T19:16:59.771 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:16:59.771 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:16:59.771 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-25T19:16:59.970 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T19:17:00.532 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 1:23:06 ago on Mon 25 Mar 2024 05:53:54 PM UTC. 2024-03-25T19:17:00.681 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-25T19:17:00.681 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T19:17:00.681 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-03-25T19:17:00.681 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T19:17:00.681 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-25T19:17:00.681 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T19:17:00.681 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:17:00.681 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-25T19:17:00.681 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T19:17:00.681 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-03-25T19:17:00.682 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:17:00.682 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 509 k 2024-03-25T19:17:00.682 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 2.5 M 2024-03-25T19:17:00.682 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-25T19:17:02.797 INFO:teuthology.orchestra.run.smithi103.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 241 kB/s | 509 kB 00:02 2024-03-25T19:17:02.798 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:17:02.798 INFO:teuthology.orchestra.run.smithi103.stdout:Total 241 kB/s | 509 kB 00:02 2024-03-25T19:17:02.798 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-25T19:17:02.802 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-25T19:17:02.802 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-25T19:17:02.842 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-25T19:17:02.843 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-25T19:17:03.005 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-25T19:17:03.104 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:17:03.533 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:17:03.810 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:17:03.810 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:17:03.810 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-25T19:17:03.810 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:17:03.810 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:17:03.810 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-25T19:17:04.020 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T19:17:04.583 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 1:23:10 ago on Mon 25 Mar 2024 05:53:54 PM UTC. 2024-03-25T19:17:04.733 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-25T19:17:04.734 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T19:17:04.734 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-03-25T19:17:04.734 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T19:17:04.734 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-25T19:17:04.734 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:17:04.734 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:17:04.734 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-25T19:17:04.734 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T19:17:04.734 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-03-25T19:17:04.734 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:17:04.735 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 52 M 2024-03-25T19:17:04.735 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 298 M 2024-03-25T19:17:04.735 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-25T19:18:25.585 INFO:teuthology.orchestra.run.smithi103.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 653 kB/s | 52 MB 01:20 2024-03-25T19:18:25.585 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:18:25.585 INFO:teuthology.orchestra.run.smithi103.stdout:Total 653 kB/s | 52 MB 01:20 2024-03-25T19:18:25.585 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-25T19:18:25.588 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-25T19:18:25.588 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-25T19:18:25.925 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-25T19:18:25.926 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-25T19:18:31.860 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-25T19:18:31.949 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:18:34.017 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:18:34.291 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:18:34.291 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:18:34.291 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-25T19:18:34.291 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:18:34.291 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:18:34.291 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-25T19:18:34.491 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T19:18:35.058 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 1:24:41 ago on Mon 25 Mar 2024 05:53:54 PM UTC. 2024-03-25T19:18:35.206 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-25T19:18:35.206 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T19:18:35.206 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-03-25T19:18:35.207 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T19:18:35.207 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-25T19:18:35.207 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T19:18:35.207 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:18:35.207 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-25T19:18:35.207 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-25T19:18:35.207 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-03-25T19:18:35.207 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:18:35.207 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 1.3 M 2024-03-25T19:18:35.208 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 5.5 M 2024-03-25T19:18:35.208 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-25T19:18:40.760 INFO:teuthology.orchestra.run.smithi103.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 231 kB/s | 1.3 MB 00:05 2024-03-25T19:18:40.761 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:18:40.762 INFO:teuthology.orchestra.run.smithi103.stdout:Total 231 kB/s | 1.3 MB 00:05 2024-03-25T19:18:40.762 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-25T19:18:40.766 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-25T19:18:40.766 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-25T19:18:40.811 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-25T19:18:40.812 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-25T19:18:41.056 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-25T19:18:41.159 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:18:41.602 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:18:41.872 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:18:41.872 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:18:41.872 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-25T19:18:41.873 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:18:41.873 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:18:41.873 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-25T19:18:42.077 DEBUG:teuthology.parallel:result is None 2024-03-25T19:18:42.078 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T19:18:42.226 DEBUG:teuthology.orchestra.run.smithi083:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T19:18:42.252 INFO:teuthology.orchestra.run.smithi083.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T19:18:42.252 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T19:18:42.252 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T19:18:42.253 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T19:18:42.398 DEBUG:teuthology.orchestra.run.smithi103:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T19:18:42.419 INFO:teuthology.orchestra.run.smithi103.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T19:18:42.420 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T19:18:42.420 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T19:18:42.421 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-25T19:18:42.421 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-25T19:18:42.421 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T19:18:42.453 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-03-25T19:18:42.453 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T19:18:42.489 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-25T19:18:42.489 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-25T19:18:42.489 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T19:18:42.523 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T19:18:42.592 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-03-25T19:18:42.592 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T19:18:42.619 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T19:18:42.686 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-25T19:18:42.686 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-25T19:18:42.687 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T19:18:42.714 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T19:18:42.782 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-03-25T19:18:42.782 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T19:18:42.809 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T19:18:42.875 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-25T19:18:42.876 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-25T19:18:42.876 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T19:18:42.903 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T19:18:42.971 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-03-25T19:18:42.971 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T19:18:42.997 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T19:18:43.064 INFO:teuthology.run_tasks:Running task ceph... 2024-03-25T19:18:43.145 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-25T19:18:43.145 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 777 /var/log/ceph 2024-03-25T19:18:43.148 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 777 /var/log/ceph 2024-03-25T19:18:43.174 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-25T19:18:43.174 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T19:18:43.216 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T19:18:43.242 INFO:tasks.ceph:Creating extra log directories... 2024-03-25T19:18:43.242 DEBUG:teuthology.orchestra.run.smithi083:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T19:18:43.283 DEBUG:teuthology.orchestra.run.smithi103:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T19:18:43.343 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-25T19:18:43.343 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-25T19:18:43.343 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621304', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7621304', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6785, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi083.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOcxMrfnS1278Hh4wj+koiCBiAB8jKgK0bfQL+LG0aENfyqJ17ESta7jqnj6tNr7rTpGvziLwO1VCqfv+Nq7aD8=', 'smithi103.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMS12b5fIyshgoPfOLpw9fDz704LM+m9FF/4ofwZhCYoyPVNHdYpMCoCpEbltvNc7n3eayx8YCzLHJb5tUCrrfs='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-25_13:28:33', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-25T19:18:43.344 DEBUG:teuthology.orchestra.run.smithi083:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T19:18:43.350 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T19:18:43.365 DEBUG:teuthology.orchestra.run.smithi083:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T19:18:43.404 DEBUG:teuthology.orchestra.run.smithi103:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T19:18:43.432 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-25T19:18:43.432 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T19:18:43.484 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T19:18:43.484 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_1 2024-03-25T19:18:43.538 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T19:18:43.538 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:18:43.538 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-25T19:18:43.538 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:18:43.538 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:18:43.538 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-03-25 17:56:36.144514418 +0000 2024-03-25T19:18:43.538 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-03-25 17:48:44.016471690 +0000 2024-03-25T19:18:43.538 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-03-25 17:48:44.016471690 +0000 2024-03-25T19:18:43.539 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: 2024-03-25 17:48:44.016471690 +0000 2024-03-25T19:18:43.539 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T19:18:43.603 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-03-25T19:18:43.603 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-03-25T19:18:43.604 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000210749 s, 2.4 MB/s 2024-03-25T19:18:43.605 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T19:18:43.661 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_2 2024-03-25T19:18:43.716 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T19:18:43.717 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:18:43.717 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-25T19:18:43.717 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:18:43.717 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:18:43.717 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-03-25 17:56:36.145514393 +0000 2024-03-25T19:18:43.717 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-03-25 17:48:44.364463232 +0000 2024-03-25T19:18:43.717 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-03-25 17:48:44.364463232 +0000 2024-03-25T19:18:43.717 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: 2024-03-25 17:48:44.364463232 +0000 2024-03-25T19:18:43.717 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T19:18:43.781 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-03-25T19:18:43.781 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-03-25T19:18:43.781 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000186297 s, 2.7 MB/s 2024-03-25T19:18:43.783 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T19:18:43.837 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_3 2024-03-25T19:18:43.892 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T19:18:43.892 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:18:43.892 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-25T19:18:43.892 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:18:43.892 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:18:43.892 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-03-25 17:56:36.145514393 +0000 2024-03-25T19:18:43.892 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-03-25 17:48:44.706454920 +0000 2024-03-25T19:18:43.892 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-03-25 17:48:44.706454920 +0000 2024-03-25T19:18:43.892 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: 2024-03-25 17:48:44.706454920 +0000 2024-03-25T19:18:43.893 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T19:18:43.957 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-03-25T19:18:43.957 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-03-25T19:18:43.957 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000180954 s, 2.8 MB/s 2024-03-25T19:18:43.958 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T19:18:44.014 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_4 2024-03-25T19:18:44.068 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T19:18:44.068 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:18:44.068 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-25T19:18:44.068 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:18:44.068 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:18:44.068 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-03-25 17:56:36.145514393 +0000 2024-03-25T19:18:44.068 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-03-25 17:48:45.038446850 +0000 2024-03-25T19:18:44.068 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-03-25 17:48:45.038446850 +0000 2024-03-25T19:18:44.068 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: 2024-03-25 17:48:45.038446850 +0000 2024-03-25T19:18:44.069 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T19:18:44.133 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-03-25T19:18:44.134 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-03-25T19:18:44.134 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.00018633 s, 2.7 MB/s 2024-03-25T19:18:44.135 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T19:18:44.190 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:18:44.190 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-03-25T19:18:44.190 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T19:18:44.208 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T19:18:44.208 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_1 2024-03-25T19:18:44.265 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T19:18:44.265 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:18:44.265 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-25T19:18:44.265 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:18:44.265 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:18:44.265 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-03-25 18:00:38.936134836 +0000 2024-03-25T19:18:44.265 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-03-25 17:48:44.017852794 +0000 2024-03-25T19:18:44.265 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-03-25 17:48:44.017852794 +0000 2024-03-25T19:18:44.265 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-03-25 17:48:44.017852794 +0000 2024-03-25T19:18:44.266 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T19:18:44.333 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-03-25T19:18:44.334 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-03-25T19:18:44.334 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000221198 s, 2.3 MB/s 2024-03-25T19:18:44.335 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T19:18:44.393 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_2 2024-03-25T19:18:44.449 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T19:18:44.449 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:18:44.449 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-25T19:18:44.449 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:18:44.450 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:18:44.450 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-03-25 18:00:38.936134836 +0000 2024-03-25T19:18:44.450 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-03-25 17:48:44.355846353 +0000 2024-03-25T19:18:44.450 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-03-25 17:48:44.355846353 +0000 2024-03-25T19:18:44.450 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-03-25 17:48:44.355846353 +0000 2024-03-25T19:18:44.450 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T19:18:44.516 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-03-25T19:18:44.516 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-03-25T19:18:44.517 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000195941 s, 2.6 MB/s 2024-03-25T19:18:44.518 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T19:18:44.575 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_3 2024-03-25T19:18:44.631 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T19:18:44.631 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:18:44.631 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-25T19:18:44.631 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:18:44.631 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:18:44.631 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-03-25 18:00:38.936134836 +0000 2024-03-25T19:18:44.631 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-03-25 17:48:44.694839893 +0000 2024-03-25T19:18:44.631 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-03-25 17:48:44.694839893 +0000 2024-03-25T19:18:44.631 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-03-25 17:48:44.694839893 +0000 2024-03-25T19:18:44.631 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T19:18:44.699 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-03-25T19:18:44.699 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-03-25T19:18:44.699 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000213527 s, 2.4 MB/s 2024-03-25T19:18:44.700 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T19:18:44.756 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_4 2024-03-25T19:18:44.811 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T19:18:44.811 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:18:44.811 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-25T19:18:44.811 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:18:44.811 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:18:44.811 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-03-25 18:00:38.937134816 +0000 2024-03-25T19:18:44.811 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-03-25 17:48:45.025833586 +0000 2024-03-25T19:18:44.812 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-03-25 17:48:45.025833586 +0000 2024-03-25T19:18:44.812 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-03-25 17:48:45.025833586 +0000 2024-03-25T19:18:44.812 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T19:18:44.877 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-03-25T19:18:44.877 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-03-25T19:18:44.877 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000167587 s, 3.1 MB/s 2024-03-25T19:18:44.878 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T19:18:44.934 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:18:44.934 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi083.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi103.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T19:18:44.934 INFO:tasks.ceph:Generating config... 2024-03-25T19:18:44.935 INFO:tasks.ceph:[client] debug monc = 20 2024-03-25T19:18:44.935 INFO:tasks.ceph:[client] debug ms = 1 2024-03-25T19:18:44.935 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-25T19:18:44.936 INFO:tasks.ceph:[client] debug rados = 20 2024-03-25T19:18:44.936 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-25T19:18:44.936 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-03-25T19:18:44.936 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-25T19:18:44.936 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-25T19:18:44.936 INFO:tasks.ceph:[global] ms type = async 2024-03-25T19:18:44.936 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-03-25T19:18:44.936 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-25T19:18:44.936 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-25T19:18:44.936 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-25T19:18:44.936 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-25T19:18:44.936 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-25T19:18:44.936 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-25T19:18:44.936 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-25T19:18:44.937 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-25T19:18:44.937 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-25T19:18:44.937 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-25T19:18:44.937 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-25T19:18:44.937 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-25T19:18:44.937 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-25T19:18:44.937 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-25T19:18:44.937 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-25T19:18:44.937 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-25T19:18:44.937 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-25T19:18:44.937 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-25T19:18:44.937 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-25T19:18:44.937 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-03-25T19:18:44.937 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-03-25T19:18:44.937 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-25T19:18:44.938 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-25T19:18:44.938 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-25T19:18:44.938 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-25T19:18:44.938 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-25T19:18:44.938 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-25T19:18:44.938 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-25T19:18:44.938 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-25T19:18:44.938 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-25T19:18:44.938 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-25T19:18:44.938 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-25T19:18:44.938 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-25T19:18:44.938 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-25T19:18:44.938 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-25T19:18:44.939 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-25T19:18:44.939 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-25T19:18:44.941 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-25T19:18:44.941 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-25T19:18:44.941 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-25T19:18:44.941 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-25T19:18:44.941 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-25T19:18:44.941 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-25T19:18:44.941 INFO:tasks.ceph:Setting up mon.a... 2024-03-25T19:18:44.942 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-25T19:18:44.999 INFO:teuthology.orchestra.run.smithi083.stdout:creating /etc/ceph/ceph.keyring 2024-03-25T19:18:45.001 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-25T19:18:45.046 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:18:45.073 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.83:6789]'), ('mon.c', '[v1:172.21.15.83:6790]'), ('mon.b', '[v1:172.21.15.103:6789]')] 2024-03-25T19:18:45.073 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.83:6789],[v1:172.21.15.83:6790],[v1:172.21.15.103:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-25T19:18:45.074 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-25T19:18:45.074 DEBUG:teuthology.orchestra.run.smithi083:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T19:18:45.129 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.83:6789]' --addv c '[v1:172.21.15.83:6790]' --addv b '[v1:172.21.15.103:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:18:45.214 INFO:teuthology.orchestra.run.smithi083.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:18:45.215 INFO:teuthology.orchestra.run.smithi083.stdout:monmaptool: generated fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:18:45.215 INFO:teuthology.orchestra.run.smithi083.stdout:setting min_mon_release = quincy 2024-03-25T19:18:45.215 INFO:teuthology.orchestra.run.smithi083.stdout:epoch 0 2024-03-25T19:18:45.215 INFO:teuthology.orchestra.run.smithi083.stdout:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:18:45.215 INFO:teuthology.orchestra.run.smithi083.stdout:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:18:45.215 INFO:teuthology.orchestra.run.smithi083.stdout:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:18:45.215 INFO:teuthology.orchestra.run.smithi083.stdout:min_mon_release 17 (quincy) 2024-03-25T19:18:45.215 INFO:teuthology.orchestra.run.smithi083.stdout:election_strategy: 3 2024-03-25T19:18:45.215 INFO:teuthology.orchestra.run.smithi083.stdout:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:18:45.215 INFO:teuthology.orchestra.run.smithi083.stdout:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:18:45.215 INFO:teuthology.orchestra.run.smithi083.stdout:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:18:45.215 INFO:teuthology.orchestra.run.smithi083.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-25T19:18:45.216 DEBUG:teuthology.orchestra.run.smithi083:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T19:18:45.231 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID e1ed2130-f46c-41bd-8514-ee278101259f... 2024-03-25T19:18:45.232 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T19:18:45.274 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T19:18:45.321 INFO:teuthology.orchestra.run.smithi083.stdout:[global] 2024-03-25T19:18:45.321 INFO:teuthology.orchestra.run.smithi083.stdout: chdir = "" 2024-03-25T19:18:45.321 INFO:teuthology.orchestra.run.smithi083.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T19:18:45.321 INFO:teuthology.orchestra.run.smithi083.stdout: auth supported = cephx 2024-03-25T19:18:45.321 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:18:45.321 INFO:teuthology.orchestra.run.smithi083.stdout: filestore xattr use omap = true 2024-03-25T19:18:45.321 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:18:45.322 INFO:teuthology.orchestra.run.smithi083.stdout: mon clock drift allowed = 1.000 2024-03-25T19:18:45.322 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:18:45.322 INFO:teuthology.orchestra.run.smithi083.stdout: osd crush chooseleaf type = 0 2024-03-25T19:18:45.322 INFO:teuthology.orchestra.run.smithi083.stdout: auth debug = true 2024-03-25T19:18:45.322 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:18:45.322 INFO:teuthology.orchestra.run.smithi083.stdout: ms die on old message = true 2024-03-25T19:18:45.322 INFO:teuthology.orchestra.run.smithi083.stdout: ms die on bug = true 2024-03-25T19:18:45.322 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:18:45.322 INFO:teuthology.orchestra.run.smithi083.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T19:18:45.322 INFO:teuthology.orchestra.run.smithi083.stdout: mon pg warn max object skew = 0 2024-03-25T19:18:45.322 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:18:45.322 INFO:teuthology.orchestra.run.smithi083.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T19:18:45.322 INFO:teuthology.orchestra.run.smithi083.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T19:18:45.323 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:18:45.323 INFO:teuthology.orchestra.run.smithi083.stdout: osd pool default size = 2 2024-03-25T19:18:45.323 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:18:45.323 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd allow primary affinity = true 2024-03-25T19:18:45.323 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd allow pg remap = true 2024-03-25T19:18:45.323 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on legacy crush tunables = false 2024-03-25T19:18:45.323 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on crush straw calc version zero = false 2024-03-25T19:18:45.323 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on no sortbitwise = false 2024-03-25T19:18:45.323 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on osd down out interval zero = false 2024-03-25T19:18:45.323 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on too few osds = false 2024-03-25T19:18:45.323 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T19:18:45.323 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T19:18:45.323 INFO:teuthology.orchestra.run.smithi083.stdout: mon_allow_pool_size_one = true 2024-03-25T19:18:45.323 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:18:45.323 INFO:teuthology.orchestra.run.smithi083.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T19:18:45.324 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:18:45.324 INFO:teuthology.orchestra.run.smithi083.stdout: osd default data pool replay window = 5 2024-03-25T19:18:45.324 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:18:45.324 INFO:teuthology.orchestra.run.smithi083.stdout: mon allow pool delete = true 2024-03-25T19:18:45.324 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:18:45.324 INFO:teuthology.orchestra.run.smithi083.stdout: mon cluster log file level = debug 2024-03-25T19:18:45.324 INFO:teuthology.orchestra.run.smithi083.stdout: debug asserts on shutdown = true 2024-03-25T19:18:45.324 INFO:teuthology.orchestra.run.smithi083.stdout: mon health detail to clog = false 2024-03-25T19:18:45.324 INFO:teuthology.orchestra.run.smithi083.stdout: mon host = "[v1:172.21.15.83:6789],[v1:172.21.15.83:6790],[v1:172.21.15.103:6789]" 2024-03-25T19:18:45.324 INFO:teuthology.orchestra.run.smithi083.stdout: mon client directed command retry = 5 2024-03-25T19:18:45.324 INFO:teuthology.orchestra.run.smithi083.stdout: mon election default strategy = 3 2024-03-25T19:18:45.324 INFO:teuthology.orchestra.run.smithi083.stdout: ms bind msgr2 = False 2024-03-25T19:18:45.324 INFO:teuthology.orchestra.run.smithi083.stdout: ms inject socket failures = 5000 2024-03-25T19:18:45.324 INFO:teuthology.orchestra.run.smithi083.stdout: ms type = async 2024-03-25T19:18:45.324 INFO:teuthology.orchestra.run.smithi083.stdout: osd heartbeat grace = 80 2024-03-25T19:18:45.324 INFO:teuthology.orchestra.run.smithi083.stdout: fsid = e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:18:45.325 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:18:45.325 INFO:teuthology.orchestra.run.smithi083.stdout:[osd] 2024-03-25T19:18:45.325 INFO:teuthology.orchestra.run.smithi083.stdout: osd journal size = 100 2024-03-25T19:18:45.325 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:18:45.325 INFO:teuthology.orchestra.run.smithi083.stdout: osd scrub load threshold = 5.0 2024-03-25T19:18:45.325 INFO:teuthology.orchestra.run.smithi083.stdout: osd scrub max interval = 600 2024-03-25T19:18:45.325 INFO:teuthology.orchestra.run.smithi083.stdout: osd mclock profile = high_recovery_ops 2024-03-25T19:18:45.325 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:18:45.325 INFO:teuthology.orchestra.run.smithi083.stdout: osd recover clone overlap = true 2024-03-25T19:18:45.325 INFO:teuthology.orchestra.run.smithi083.stdout: osd recovery max chunk = 1048576 2024-03-25T19:18:45.325 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:18:45.325 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug shutdown = true 2024-03-25T19:18:45.325 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug op order = true 2024-03-25T19:18:45.325 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug verify stray on activate = true 2024-03-25T19:18:45.325 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug trim objects = true 2024-03-25T19:18:45.325 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:18:45.326 INFO:teuthology.orchestra.run.smithi083.stdout: osd open classes on start = true 2024-03-25T19:18:45.326 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug pg log writeout = true 2024-03-25T19:18:45.326 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:18:45.326 INFO:teuthology.orchestra.run.smithi083.stdout: osd deep scrub update digest min age = 30 2024-03-25T19:18:45.326 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:18:45.326 INFO:teuthology.orchestra.run.smithi083.stdout: osd map max advance = 10 2024-03-25T19:18:45.326 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:18:45.326 INFO:teuthology.orchestra.run.smithi083.stdout: journal zero on create = true 2024-03-25T19:18:45.326 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:18:45.326 INFO:teuthology.orchestra.run.smithi083.stdout: filestore ondisk finisher threads = 3 2024-03-25T19:18:45.326 INFO:teuthology.orchestra.run.smithi083.stdout: filestore apply finisher threads = 3 2024-03-25T19:18:45.326 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:18:45.326 INFO:teuthology.orchestra.run.smithi083.stdout: bdev debug aio = true 2024-03-25T19:18:45.326 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug misdirected ops = true 2024-03-25T19:18:45.326 INFO:teuthology.orchestra.run.smithi083.stdout: bdev async discard = True 2024-03-25T19:18:45.327 INFO:teuthology.orchestra.run.smithi083.stdout: bdev enable discard = True 2024-03-25T19:18:45.327 INFO:teuthology.orchestra.run.smithi083.stdout: bluestore allocator = stupid 2024-03-25T19:18:45.327 INFO:teuthology.orchestra.run.smithi083.stdout: bluestore block size = 96636764160 2024-03-25T19:18:45.327 INFO:teuthology.orchestra.run.smithi083.stdout: bluestore fsck on mount = True 2024-03-25T19:18:45.327 INFO:teuthology.orchestra.run.smithi083.stdout: bluestore zero block detection = True 2024-03-25T19:18:45.327 INFO:teuthology.orchestra.run.smithi083.stdout: debug bluefs = 1 2024-03-25T19:18:45.327 INFO:teuthology.orchestra.run.smithi083.stdout: debug bluestore = 1 2024-03-25T19:18:45.327 INFO:teuthology.orchestra.run.smithi083.stdout: debug monc = 20 2024-03-25T19:18:45.327 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-03-25T19:18:45.327 INFO:teuthology.orchestra.run.smithi083.stdout: debug osd = 20 2024-03-25T19:18:45.327 INFO:teuthology.orchestra.run.smithi083.stdout: debug rocksdb = 10 2024-03-25T19:18:45.327 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T19:18:45.327 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd full ratio = 0.9 2024-03-25T19:18:45.327 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd nearfull ratio = 0.8 2024-03-25T19:18:45.327 INFO:teuthology.orchestra.run.smithi083.stdout: osd blocked scrub grace period = 3600 2024-03-25T19:18:45.327 INFO:teuthology.orchestra.run.smithi083.stdout: osd class default list = * 2024-03-25T19:18:45.327 INFO:teuthology.orchestra.run.smithi083.stdout: osd class load list = * 2024-03-25T19:18:45.328 INFO:teuthology.orchestra.run.smithi083.stdout: osd client watch timeout = 120 2024-03-25T19:18:45.328 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug verify cached snaps = True 2024-03-25T19:18:45.328 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug verify missing on start = True 2024-03-25T19:18:45.328 INFO:teuthology.orchestra.run.smithi083.stdout: osd failsafe full ratio = 0.95 2024-03-25T19:18:45.328 INFO:teuthology.orchestra.run.smithi083.stdout: osd fast shutdown = False 2024-03-25T19:18:45.328 INFO:teuthology.orchestra.run.smithi083.stdout: osd max markdown count = 1000 2024-03-25T19:18:45.328 INFO:teuthology.orchestra.run.smithi083.stdout: osd mclock override recovery settings = True 2024-03-25T19:18:45.328 INFO:teuthology.orchestra.run.smithi083.stdout: osd objectstore = bluestore 2024-03-25T19:18:45.328 INFO:teuthology.orchestra.run.smithi083.stdout: osd op queue = debug_random 2024-03-25T19:18:45.328 INFO:teuthology.orchestra.run.smithi083.stdout: osd op queue cut off = debug_random 2024-03-25T19:18:45.328 INFO:teuthology.orchestra.run.smithi083.stdout: osd shutdown pgref assert = True 2024-03-25T19:18:45.328 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:18:45.328 INFO:teuthology.orchestra.run.smithi083.stdout:[mgr] 2024-03-25T19:18:45.328 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-03-25T19:18:45.328 INFO:teuthology.orchestra.run.smithi083.stdout: debug mgr = 20 2024-03-25T19:18:45.328 INFO:teuthology.orchestra.run.smithi083.stdout: debug mon = 20 2024-03-25T19:18:45.329 INFO:teuthology.orchestra.run.smithi083.stdout: debug auth = 20 2024-03-25T19:18:45.329 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:18:45.329 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:18:45.329 INFO:teuthology.orchestra.run.smithi083.stdout: mgr/telemetry/nag = false 2024-03-25T19:18:45.329 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:18:45.329 INFO:teuthology.orchestra.run.smithi083.stdout:[mon] 2024-03-25T19:18:45.329 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-03-25T19:18:45.329 INFO:teuthology.orchestra.run.smithi083.stdout: debug mon = 20 2024-03-25T19:18:45.329 INFO:teuthology.orchestra.run.smithi083.stdout: debug paxos = 20 2024-03-25T19:18:45.329 INFO:teuthology.orchestra.run.smithi083.stdout: debug auth = 20 2024-03-25T19:18:45.329 INFO:teuthology.orchestra.run.smithi083.stdout: mon data avail warn = 5 2024-03-25T19:18:45.329 INFO:teuthology.orchestra.run.smithi083.stdout: mon mgr mkfs grace = 240 2024-03-25T19:18:45.329 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:18:45.329 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd reporter subtree level = osd 2024-03-25T19:18:45.329 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd prime pg temp = true 2024-03-25T19:18:45.330 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:18:45.330 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:18:45.330 INFO:teuthology.orchestra.run.smithi083.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T19:18:45.330 INFO:teuthology.orchestra.run.smithi083.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T19:18:45.330 INFO:teuthology.orchestra.run.smithi083.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T19:18:45.330 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:18:45.330 INFO:teuthology.orchestra.run.smithi083.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T19:18:45.330 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T19:18:45.330 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T19:18:45.330 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:18:45.330 INFO:teuthology.orchestra.run.smithi083.stdout: # 1m isn't quite enough 2024-03-25T19:18:45.330 INFO:teuthology.orchestra.run.smithi083.stdout: mon_down_mkfs_grace = 2m 2024-03-25T19:18:45.330 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:18:45.330 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_filestore_osds = false 2024-03-25T19:18:45.330 INFO:teuthology.orchestra.run.smithi083.stdout: mon min osdmap epochs = 50 2024-03-25T19:18:45.331 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd crush smoke test = False 2024-03-25T19:18:45.331 INFO:teuthology.orchestra.run.smithi083.stdout: mon osdmap full prune interval = 2 2024-03-25T19:18:45.331 INFO:teuthology.orchestra.run.smithi083.stdout: mon osdmap full prune min = 15 2024-03-25T19:18:45.331 INFO:teuthology.orchestra.run.smithi083.stdout: mon osdmap full prune txsize = 2 2024-03-25T19:18:45.331 INFO:teuthology.orchestra.run.smithi083.stdout: mon scrub interval = 300 2024-03-25T19:18:45.331 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on pool no app = False 2024-03-25T19:18:45.331 INFO:teuthology.orchestra.run.smithi083.stdout: paxos service trim min = 10 2024-03-25T19:18:45.331 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:18:45.331 INFO:teuthology.orchestra.run.smithi083.stdout:[client] 2024-03-25T19:18:45.331 INFO:teuthology.orchestra.run.smithi083.stdout: rgw cache enabled = true 2024-03-25T19:18:45.331 INFO:teuthology.orchestra.run.smithi083.stdout: rgw enable ops log = true 2024-03-25T19:18:45.331 INFO:teuthology.orchestra.run.smithi083.stdout: rgw enable usage log = true 2024-03-25T19:18:45.331 INFO:teuthology.orchestra.run.smithi083.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T19:18:45.331 INFO:teuthology.orchestra.run.smithi083.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T19:18:45.331 INFO:teuthology.orchestra.run.smithi083.stdout: debug monc = 20 2024-03-25T19:18:45.332 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-03-25T19:18:45.332 INFO:teuthology.orchestra.run.smithi083.stdout: debug objecter = 20 2024-03-25T19:18:45.332 INFO:teuthology.orchestra.run.smithi083.stdout: debug rados = 20 2024-03-25T19:18:45.332 INFO:teuthology.orchestra.run.smithi083.stdout:[mon.a] 2024-03-25T19:18:45.332 INFO:teuthology.orchestra.run.smithi083.stdout:[mon.c] 2024-03-25T19:18:45.332 INFO:teuthology.orchestra.run.smithi083.stdout:[mon.b] 2024-03-25T19:18:45.332 INFO:teuthology.orchestra.run.smithi103.stdout:[global] 2024-03-25T19:18:45.332 INFO:teuthology.orchestra.run.smithi103.stdout: chdir = "" 2024-03-25T19:18:45.332 INFO:teuthology.orchestra.run.smithi103.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T19:18:45.332 INFO:teuthology.orchestra.run.smithi103.stdout: auth supported = cephx 2024-03-25T19:18:45.333 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:18:45.333 INFO:teuthology.orchestra.run.smithi103.stdout: filestore xattr use omap = true 2024-03-25T19:18:45.333 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:18:45.333 INFO:teuthology.orchestra.run.smithi103.stdout: mon clock drift allowed = 1.000 2024-03-25T19:18:45.333 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:18:45.333 INFO:teuthology.orchestra.run.smithi103.stdout: osd crush chooseleaf type = 0 2024-03-25T19:18:45.333 INFO:teuthology.orchestra.run.smithi103.stdout: auth debug = true 2024-03-25T19:18:45.333 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:18:45.333 INFO:teuthology.orchestra.run.smithi103.stdout: ms die on old message = true 2024-03-25T19:18:45.333 INFO:teuthology.orchestra.run.smithi103.stdout: ms die on bug = true 2024-03-25T19:18:45.333 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:18:45.333 INFO:teuthology.orchestra.run.smithi103.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T19:18:45.333 INFO:teuthology.orchestra.run.smithi103.stdout: mon pg warn max object skew = 0 2024-03-25T19:18:45.333 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:18:45.333 INFO:teuthology.orchestra.run.smithi103.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T19:18:45.334 INFO:teuthology.orchestra.run.smithi103.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T19:18:45.334 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:18:45.334 INFO:teuthology.orchestra.run.smithi103.stdout: osd pool default size = 2 2024-03-25T19:18:45.334 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:18:45.334 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd allow primary affinity = true 2024-03-25T19:18:45.334 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd allow pg remap = true 2024-03-25T19:18:45.334 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on legacy crush tunables = false 2024-03-25T19:18:45.334 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on crush straw calc version zero = false 2024-03-25T19:18:45.334 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on no sortbitwise = false 2024-03-25T19:18:45.334 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on osd down out interval zero = false 2024-03-25T19:18:45.334 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on too few osds = false 2024-03-25T19:18:45.334 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T19:18:45.334 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T19:18:45.334 INFO:teuthology.orchestra.run.smithi103.stdout: mon_allow_pool_size_one = true 2024-03-25T19:18:45.335 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:18:45.335 INFO:teuthology.orchestra.run.smithi103.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T19:18:45.335 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:18:45.335 INFO:teuthology.orchestra.run.smithi103.stdout: osd default data pool replay window = 5 2024-03-25T19:18:45.335 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:18:45.335 INFO:teuthology.orchestra.run.smithi103.stdout: mon allow pool delete = true 2024-03-25T19:18:45.335 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:18:45.335 INFO:teuthology.orchestra.run.smithi103.stdout: mon cluster log file level = debug 2024-03-25T19:18:45.335 INFO:teuthology.orchestra.run.smithi103.stdout: debug asserts on shutdown = true 2024-03-25T19:18:45.335 INFO:teuthology.orchestra.run.smithi103.stdout: mon health detail to clog = false 2024-03-25T19:18:45.335 INFO:teuthology.orchestra.run.smithi103.stdout: mon host = "[v1:172.21.15.83:6789],[v1:172.21.15.83:6790],[v1:172.21.15.103:6789]" 2024-03-25T19:18:45.335 INFO:teuthology.orchestra.run.smithi103.stdout: mon client directed command retry = 5 2024-03-25T19:18:45.335 INFO:teuthology.orchestra.run.smithi103.stdout: mon election default strategy = 3 2024-03-25T19:18:45.335 INFO:teuthology.orchestra.run.smithi103.stdout: ms bind msgr2 = False 2024-03-25T19:18:45.335 INFO:teuthology.orchestra.run.smithi103.stdout: ms inject socket failures = 5000 2024-03-25T19:18:45.335 INFO:teuthology.orchestra.run.smithi103.stdout: ms type = async 2024-03-25T19:18:45.336 INFO:teuthology.orchestra.run.smithi103.stdout: osd heartbeat grace = 80 2024-03-25T19:18:45.336 INFO:teuthology.orchestra.run.smithi103.stdout: fsid = e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:18:45.336 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:18:45.336 INFO:teuthology.orchestra.run.smithi103.stdout:[osd] 2024-03-25T19:18:45.336 INFO:teuthology.orchestra.run.smithi103.stdout: osd journal size = 100 2024-03-25T19:18:45.336 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:18:45.336 INFO:teuthology.orchestra.run.smithi103.stdout: osd scrub load threshold = 5.0 2024-03-25T19:18:45.336 INFO:teuthology.orchestra.run.smithi103.stdout: osd scrub max interval = 600 2024-03-25T19:18:45.336 INFO:teuthology.orchestra.run.smithi103.stdout: osd mclock profile = high_recovery_ops 2024-03-25T19:18:45.336 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:18:45.336 INFO:teuthology.orchestra.run.smithi103.stdout: osd recover clone overlap = true 2024-03-25T19:18:45.336 INFO:teuthology.orchestra.run.smithi103.stdout: osd recovery max chunk = 1048576 2024-03-25T19:18:45.336 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:18:45.336 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug shutdown = true 2024-03-25T19:18:45.336 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug op order = true 2024-03-25T19:18:45.336 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug verify stray on activate = true 2024-03-25T19:18:45.336 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug trim objects = true 2024-03-25T19:18:45.337 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:18:45.337 INFO:teuthology.orchestra.run.smithi103.stdout: osd open classes on start = true 2024-03-25T19:18:45.337 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug pg log writeout = true 2024-03-25T19:18:45.337 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:18:45.337 INFO:teuthology.orchestra.run.smithi103.stdout: osd deep scrub update digest min age = 30 2024-03-25T19:18:45.337 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:18:45.337 INFO:teuthology.orchestra.run.smithi103.stdout: osd map max advance = 10 2024-03-25T19:18:45.337 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:18:45.337 INFO:teuthology.orchestra.run.smithi103.stdout: journal zero on create = true 2024-03-25T19:18:45.337 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:18:45.337 INFO:teuthology.orchestra.run.smithi103.stdout: filestore ondisk finisher threads = 3 2024-03-25T19:18:45.337 INFO:teuthology.orchestra.run.smithi103.stdout: filestore apply finisher threads = 3 2024-03-25T19:18:45.337 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:18:45.337 INFO:teuthology.orchestra.run.smithi103.stdout: bdev debug aio = true 2024-03-25T19:18:45.337 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug misdirected ops = true 2024-03-25T19:18:45.338 INFO:teuthology.orchestra.run.smithi103.stdout: bdev async discard = True 2024-03-25T19:18:45.338 INFO:teuthology.orchestra.run.smithi103.stdout: bdev enable discard = True 2024-03-25T19:18:45.338 INFO:teuthology.orchestra.run.smithi103.stdout: bluestore allocator = stupid 2024-03-25T19:18:45.338 INFO:teuthology.orchestra.run.smithi103.stdout: bluestore block size = 96636764160 2024-03-25T19:18:45.338 INFO:teuthology.orchestra.run.smithi103.stdout: bluestore fsck on mount = True 2024-03-25T19:18:45.338 INFO:teuthology.orchestra.run.smithi103.stdout: bluestore zero block detection = True 2024-03-25T19:18:45.338 INFO:teuthology.orchestra.run.smithi103.stdout: debug bluefs = 1 2024-03-25T19:18:45.338 INFO:teuthology.orchestra.run.smithi103.stdout: debug bluestore = 1 2024-03-25T19:18:45.338 INFO:teuthology.orchestra.run.smithi103.stdout: debug monc = 20 2024-03-25T19:18:45.338 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2024-03-25T19:18:45.338 INFO:teuthology.orchestra.run.smithi103.stdout: debug osd = 20 2024-03-25T19:18:45.338 INFO:teuthology.orchestra.run.smithi103.stdout: debug rocksdb = 10 2024-03-25T19:18:45.338 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T19:18:45.338 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd full ratio = 0.9 2024-03-25T19:18:45.338 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd nearfull ratio = 0.8 2024-03-25T19:18:45.339 INFO:teuthology.orchestra.run.smithi103.stdout: osd blocked scrub grace period = 3600 2024-03-25T19:18:45.339 INFO:teuthology.orchestra.run.smithi103.stdout: osd class default list = * 2024-03-25T19:18:45.339 INFO:teuthology.orchestra.run.smithi103.stdout: osd class load list = * 2024-03-25T19:18:45.339 INFO:teuthology.orchestra.run.smithi103.stdout: osd client watch timeout = 120 2024-03-25T19:18:45.339 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug verify cached snaps = True 2024-03-25T19:18:45.339 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug verify missing on start = True 2024-03-25T19:18:45.339 INFO:teuthology.orchestra.run.smithi103.stdout: osd failsafe full ratio = 0.95 2024-03-25T19:18:45.339 INFO:teuthology.orchestra.run.smithi103.stdout: osd fast shutdown = False 2024-03-25T19:18:45.339 INFO:teuthology.orchestra.run.smithi103.stdout: osd max markdown count = 1000 2024-03-25T19:18:45.339 INFO:teuthology.orchestra.run.smithi103.stdout: osd mclock override recovery settings = True 2024-03-25T19:18:45.339 INFO:teuthology.orchestra.run.smithi103.stdout: osd objectstore = bluestore 2024-03-25T19:18:45.339 INFO:teuthology.orchestra.run.smithi103.stdout: osd op queue = debug_random 2024-03-25T19:18:45.339 INFO:teuthology.orchestra.run.smithi103.stdout: osd op queue cut off = debug_random 2024-03-25T19:18:45.339 INFO:teuthology.orchestra.run.smithi103.stdout: osd shutdown pgref assert = True 2024-03-25T19:18:45.340 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:18:45.340 INFO:teuthology.orchestra.run.smithi103.stdout:[mgr] 2024-03-25T19:18:45.340 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2024-03-25T19:18:45.340 INFO:teuthology.orchestra.run.smithi103.stdout: debug mgr = 20 2024-03-25T19:18:45.340 INFO:teuthology.orchestra.run.smithi103.stdout: debug mon = 20 2024-03-25T19:18:45.340 INFO:teuthology.orchestra.run.smithi103.stdout: debug auth = 20 2024-03-25T19:18:45.340 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:18:45.340 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:18:45.340 INFO:teuthology.orchestra.run.smithi103.stdout: mgr/telemetry/nag = false 2024-03-25T19:18:45.340 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:18:45.340 INFO:teuthology.orchestra.run.smithi103.stdout:[mon] 2024-03-25T19:18:45.340 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2024-03-25T19:18:45.340 INFO:teuthology.orchestra.run.smithi103.stdout: debug mon = 20 2024-03-25T19:18:45.340 INFO:teuthology.orchestra.run.smithi103.stdout: debug paxos = 20 2024-03-25T19:18:45.340 INFO:teuthology.orchestra.run.smithi103.stdout: debug auth = 20 2024-03-25T19:18:45.341 INFO:teuthology.orchestra.run.smithi103.stdout: mon data avail warn = 5 2024-03-25T19:18:45.341 INFO:teuthology.orchestra.run.smithi103.stdout: mon mgr mkfs grace = 240 2024-03-25T19:18:45.341 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:18:45.341 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd reporter subtree level = osd 2024-03-25T19:18:45.341 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd prime pg temp = true 2024-03-25T19:18:45.341 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:18:45.341 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:18:45.341 INFO:teuthology.orchestra.run.smithi103.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T19:18:45.341 INFO:teuthology.orchestra.run.smithi103.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T19:18:45.341 INFO:teuthology.orchestra.run.smithi103.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T19:18:45.341 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:18:45.341 INFO:teuthology.orchestra.run.smithi103.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T19:18:45.341 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T19:18:45.341 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T19:18:45.341 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:18:45.342 INFO:teuthology.orchestra.run.smithi103.stdout: # 1m isn't quite enough 2024-03-25T19:18:45.342 INFO:teuthology.orchestra.run.smithi103.stdout: mon_down_mkfs_grace = 2m 2024-03-25T19:18:45.342 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:18:45.342 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_filestore_osds = false 2024-03-25T19:18:45.342 INFO:teuthology.orchestra.run.smithi103.stdout: mon min osdmap epochs = 50 2024-03-25T19:18:45.342 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd crush smoke test = False 2024-03-25T19:18:45.342 INFO:teuthology.orchestra.run.smithi103.stdout: mon osdmap full prune interval = 2 2024-03-25T19:18:45.342 INFO:teuthology.orchestra.run.smithi103.stdout: mon osdmap full prune min = 15 2024-03-25T19:18:45.342 INFO:teuthology.orchestra.run.smithi103.stdout: mon osdmap full prune txsize = 2 2024-03-25T19:18:45.342 INFO:teuthology.orchestra.run.smithi103.stdout: mon scrub interval = 300 2024-03-25T19:18:45.342 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on pool no app = False 2024-03-25T19:18:45.342 INFO:teuthology.orchestra.run.smithi103.stdout: paxos service trim min = 10 2024-03-25T19:18:45.342 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-25T19:18:45.342 INFO:teuthology.orchestra.run.smithi103.stdout:[client] 2024-03-25T19:18:45.342 INFO:teuthology.orchestra.run.smithi103.stdout: rgw cache enabled = true 2024-03-25T19:18:45.343 INFO:teuthology.orchestra.run.smithi103.stdout: rgw enable ops log = true 2024-03-25T19:18:45.343 INFO:teuthology.orchestra.run.smithi103.stdout: rgw enable usage log = true 2024-03-25T19:18:45.343 INFO:teuthology.orchestra.run.smithi103.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T19:18:45.343 INFO:teuthology.orchestra.run.smithi103.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T19:18:45.343 INFO:teuthology.orchestra.run.smithi103.stdout: debug monc = 20 2024-03-25T19:18:45.343 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2024-03-25T19:18:45.343 INFO:teuthology.orchestra.run.smithi103.stdout: debug objecter = 20 2024-03-25T19:18:45.343 INFO:teuthology.orchestra.run.smithi103.stdout: debug rados = 20 2024-03-25T19:18:45.343 INFO:teuthology.orchestra.run.smithi103.stdout:[mon.a] 2024-03-25T19:18:45.343 INFO:teuthology.orchestra.run.smithi103.stdout:[mon.c] 2024-03-25T19:18:45.343 INFO:teuthology.orchestra.run.smithi103.stdout:[mon.b] 2024-03-25T19:18:45.344 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-25T19:18:45.345 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-25T19:18:45.399 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-25T19:18:45.399 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-25T19:18:45.399 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-25T19:18:45.414 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-25T19:18:45.414 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-25T19:18:45.468 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi083.front.sepia.ceph.com 2024-03-25T19:18:45.468 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-25T19:18:45.468 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T19:18:45.468 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:18:45.547 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-25T19:18:45.547 DEBUG:teuthology.orchestra.run.smithi083:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:18:45.602 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi103.front.sepia.ceph.com 2024-03-25T19:18:45.602 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-03-25T19:18:45.602 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T19:18:45.602 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:18:45.642 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-03-25T19:18:45.642 DEBUG:teuthology.orchestra.run.smithi103:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:18:45.696 INFO:tasks.ceph:Setting up mon nodes... 2024-03-25T19:18:45.697 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-25T19:18:45.697 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T19:18:45.751 INFO:teuthology.orchestra.run.smithi083.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T19:18:45.753 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T19:18:45.832 INFO:teuthology.orchestra.run.smithi103.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T19:18:45.834 INFO:tasks.ceph:Setting up mds nodes... 2024-03-25T19:18:45.834 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-25T19:18:45.834 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-25T19:18:45.876 INFO:teuthology.orchestra.run.smithi083.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-25T19:18:45.891 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-25T19:18:45.933 INFO:teuthology.orchestra.run.smithi103.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-25T19:18:45.948 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-25T19:18:45.948 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi083.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi103.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T19:18:45.948 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-25T19:18:45.975 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:18:45.975 INFO:tasks.ceph:role: osd.0 2024-03-25T19:18:45.975 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi083.front.sepia.ceph.com 2024-03-25T19:18:45.975 DEBUG:teuthology.orchestra.run.smithi083:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T19:18:46.049 INFO:teuthology.orchestra.run.smithi083.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:18:46.049 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:18:46.049 INFO:teuthology.orchestra.run.smithi083.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:18:46.050 INFO:teuthology.orchestra.run.smithi083.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:18:46.050 INFO:teuthology.orchestra.run.smithi083.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:18:46.050 INFO:teuthology.orchestra.run.smithi083.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:18:46.050 INFO:teuthology.orchestra.run.smithi083.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:18:46.050 INFO:teuthology.orchestra.run.smithi083.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:18:46.050 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:18:46.050 INFO:teuthology.orchestra.run.smithi083.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:18:46.565 INFO:teuthology.orchestra.run.smithi083.stdout:Discarding blocks...Done. 2024-03-25T19:18:46.567 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi083.front.sepia.ceph.com -o noatime 2024-03-25T19:18:46.567 DEBUG:teuthology.orchestra.run.smithi083:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-25T19:18:46.718 DEBUG:teuthology.orchestra.run.smithi083:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-25T19:18:46.752 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-25T19:18:46.820 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:18:46.820 INFO:tasks.ceph:role: osd.1 2024-03-25T19:18:46.820 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi083.front.sepia.ceph.com 2024-03-25T19:18:46.820 DEBUG:teuthology.orchestra.run.smithi083:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T19:18:46.894 INFO:teuthology.orchestra.run.smithi083.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:18:46.894 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:18:46.894 INFO:teuthology.orchestra.run.smithi083.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:18:46.894 INFO:teuthology.orchestra.run.smithi083.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:18:46.894 INFO:teuthology.orchestra.run.smithi083.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:18:46.894 INFO:teuthology.orchestra.run.smithi083.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:18:46.894 INFO:teuthology.orchestra.run.smithi083.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:18:46.894 INFO:teuthology.orchestra.run.smithi083.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:18:46.894 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:18:46.895 INFO:teuthology.orchestra.run.smithi083.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:18:47.336 INFO:teuthology.orchestra.run.smithi083.stdout:Discarding blocks...Done. 2024-03-25T19:18:47.338 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi083.front.sepia.ceph.com -o noatime 2024-03-25T19:18:47.338 DEBUG:teuthology.orchestra.run.smithi083:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-25T19:18:47.432 DEBUG:teuthology.orchestra.run.smithi083:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-25T19:18:47.465 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-25T19:18:47.532 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:18:47.532 INFO:tasks.ceph:role: osd.2 2024-03-25T19:18:47.532 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi083.front.sepia.ceph.com 2024-03-25T19:18:47.532 DEBUG:teuthology.orchestra.run.smithi083:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T19:18:47.605 INFO:teuthology.orchestra.run.smithi083.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:18:47.606 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:18:47.606 INFO:teuthology.orchestra.run.smithi083.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:18:47.606 INFO:teuthology.orchestra.run.smithi083.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:18:47.606 INFO:teuthology.orchestra.run.smithi083.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:18:47.606 INFO:teuthology.orchestra.run.smithi083.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:18:47.606 INFO:teuthology.orchestra.run.smithi083.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:18:47.606 INFO:teuthology.orchestra.run.smithi083.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:18:47.606 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:18:47.606 INFO:teuthology.orchestra.run.smithi083.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:18:48.087 INFO:teuthology.orchestra.run.smithi083.stdout:Discarding blocks...Done. 2024-03-25T19:18:48.089 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi083.front.sepia.ceph.com -o noatime 2024-03-25T19:18:48.089 DEBUG:teuthology.orchestra.run.smithi083:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-25T19:18:48.211 DEBUG:teuthology.orchestra.run.smithi083:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-25T19:18:48.243 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-25T19:18:48.310 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:18:48.310 INFO:tasks.ceph:role: osd.3 2024-03-25T19:18:48.310 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi083.front.sepia.ceph.com 2024-03-25T19:18:48.310 DEBUG:teuthology.orchestra.run.smithi083:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T19:18:48.384 INFO:teuthology.orchestra.run.smithi083.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:18:48.385 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:18:48.385 INFO:teuthology.orchestra.run.smithi083.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:18:48.385 INFO:teuthology.orchestra.run.smithi083.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:18:48.385 INFO:teuthology.orchestra.run.smithi083.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:18:48.385 INFO:teuthology.orchestra.run.smithi083.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:18:48.385 INFO:teuthology.orchestra.run.smithi083.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:18:48.385 INFO:teuthology.orchestra.run.smithi083.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:18:48.385 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:18:48.385 INFO:teuthology.orchestra.run.smithi083.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:18:48.896 INFO:teuthology.orchestra.run.smithi083.stdout:Discarding blocks...Done. 2024-03-25T19:18:48.898 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi083.front.sepia.ceph.com -o noatime 2024-03-25T19:18:48.898 DEBUG:teuthology.orchestra.run.smithi083:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-25T19:18:49.016 DEBUG:teuthology.orchestra.run.smithi083:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-25T19:18:49.050 DEBUG:teuthology.orchestra.run.smithi083:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:18:49.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:18:49.156+0000 7f564459a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-25T19:18:49.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:18:49.156+0000 7f564459a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-25T19:18:49.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:18:49.156+0000 7f564459a640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:18:49.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:18:49.156+0000 7f564459a640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:18:49.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:18:49.157+0000 7f564459a640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:18:49.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:18:49.157+0000 7f564459a640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-25T19:18:52.003 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-25T19:18:52.033 DEBUG:teuthology.orchestra.run.smithi083:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:18:52.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:18:52.123+0000 7fe4d4c19640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-25T19:18:52.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:18:52.123+0000 7fe4d4c19640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-25T19:18:52.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:18:52.123+0000 7fe4d4c19640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:18:52.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:18:52.123+0000 7fe4d4c19640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:18:52.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:18:52.123+0000 7fe4d4c19640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:18:52.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:18:52.124+0000 7fe4d4c19640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-25T19:18:54.961 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-25T19:18:54.992 DEBUG:teuthology.orchestra.run.smithi083:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:18:55.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:18:55.082+0000 7effa2b8f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-25T19:18:55.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:18:55.083+0000 7effa2b8f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-25T19:18:55.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:18:55.083+0000 7effa2b8f640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:18:55.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:18:55.083+0000 7effa2b8f640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:18:55.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:18:55.083+0000 7effa2b8f640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:18:55.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:18:55.083+0000 7effa2b8f640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-25T19:18:57.932 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-25T19:18:57.964 DEBUG:teuthology.orchestra.run.smithi083:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:18:58.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:18:58.054+0000 7fbaa5413640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-25T19:18:58.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:18:58.054+0000 7fbaa5413640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-25T19:18:58.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:18:58.054+0000 7fbaa5413640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:18:58.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:18:58.054+0000 7fbaa5413640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:18:58.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:18:58.054+0000 7fbaa5413640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:18:58.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:18:58.055+0000 7fbaa5413640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-25T19:19:00.897 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-25T19:19:00.928 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-25T19:19:00.960 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:19:00.960 INFO:tasks.ceph:role: osd.4 2024-03-25T19:19:00.960 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi103.front.sepia.ceph.com 2024-03-25T19:19:00.960 DEBUG:teuthology.orchestra.run.smithi103:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T19:19:01.038 INFO:teuthology.orchestra.run.smithi103.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:19:01.038 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:19:01.038 INFO:teuthology.orchestra.run.smithi103.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:19:01.038 INFO:teuthology.orchestra.run.smithi103.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:19:01.038 INFO:teuthology.orchestra.run.smithi103.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:19:01.039 INFO:teuthology.orchestra.run.smithi103.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:19:01.039 INFO:teuthology.orchestra.run.smithi103.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:19:01.039 INFO:teuthology.orchestra.run.smithi103.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:19:01.039 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:19:01.039 INFO:teuthology.orchestra.run.smithi103.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:19:01.553 INFO:teuthology.orchestra.run.smithi103.stdout:Discarding blocks...Done. 2024-03-25T19:19:01.555 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi103.front.sepia.ceph.com -o noatime 2024-03-25T19:19:01.555 DEBUG:teuthology.orchestra.run.smithi103:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-25T19:19:01.703 DEBUG:teuthology.orchestra.run.smithi103:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-25T19:19:01.738 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-25T19:19:01.807 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:19:01.807 INFO:tasks.ceph:role: osd.5 2024-03-25T19:19:01.807 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi103.front.sepia.ceph.com 2024-03-25T19:19:01.807 DEBUG:teuthology.orchestra.run.smithi103:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T19:19:01.882 INFO:teuthology.orchestra.run.smithi103.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:19:01.882 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:19:01.882 INFO:teuthology.orchestra.run.smithi103.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:19:01.882 INFO:teuthology.orchestra.run.smithi103.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:19:01.882 INFO:teuthology.orchestra.run.smithi103.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:19:01.882 INFO:teuthology.orchestra.run.smithi103.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:19:01.882 INFO:teuthology.orchestra.run.smithi103.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:19:01.882 INFO:teuthology.orchestra.run.smithi103.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:19:01.882 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:19:01.883 INFO:teuthology.orchestra.run.smithi103.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:19:02.331 INFO:teuthology.orchestra.run.smithi103.stdout:Discarding blocks...Done. 2024-03-25T19:19:02.333 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi103.front.sepia.ceph.com -o noatime 2024-03-25T19:19:02.333 DEBUG:teuthology.orchestra.run.smithi103:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-25T19:19:02.433 DEBUG:teuthology.orchestra.run.smithi103:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-25T19:19:02.467 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-25T19:19:02.537 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:19:02.537 INFO:tasks.ceph:role: osd.6 2024-03-25T19:19:02.537 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi103.front.sepia.ceph.com 2024-03-25T19:19:02.537 DEBUG:teuthology.orchestra.run.smithi103:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T19:19:02.613 INFO:teuthology.orchestra.run.smithi103.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:19:02.613 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:19:02.613 INFO:teuthology.orchestra.run.smithi103.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:19:02.613 INFO:teuthology.orchestra.run.smithi103.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:19:02.613 INFO:teuthology.orchestra.run.smithi103.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:19:02.614 INFO:teuthology.orchestra.run.smithi103.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:19:02.614 INFO:teuthology.orchestra.run.smithi103.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:19:02.614 INFO:teuthology.orchestra.run.smithi103.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:19:02.614 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:19:02.614 INFO:teuthology.orchestra.run.smithi103.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:19:03.098 INFO:teuthology.orchestra.run.smithi103.stdout:Discarding blocks...Done. 2024-03-25T19:19:03.100 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi103.front.sepia.ceph.com -o noatime 2024-03-25T19:19:03.100 DEBUG:teuthology.orchestra.run.smithi103:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-25T19:19:03.234 DEBUG:teuthology.orchestra.run.smithi103:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-25T19:19:03.270 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-25T19:19:03.340 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:19:03.340 INFO:tasks.ceph:role: osd.7 2024-03-25T19:19:03.340 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi103.front.sepia.ceph.com 2024-03-25T19:19:03.340 DEBUG:teuthology.orchestra.run.smithi103:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T19:19:03.416 INFO:teuthology.orchestra.run.smithi103.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:19:03.416 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:19:03.416 INFO:teuthology.orchestra.run.smithi103.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:19:03.416 INFO:teuthology.orchestra.run.smithi103.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:19:03.416 INFO:teuthology.orchestra.run.smithi103.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:19:03.416 INFO:teuthology.orchestra.run.smithi103.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:19:03.417 INFO:teuthology.orchestra.run.smithi103.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:19:03.417 INFO:teuthology.orchestra.run.smithi103.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:19:03.417 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:19:03.417 INFO:teuthology.orchestra.run.smithi103.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:19:03.909 INFO:teuthology.orchestra.run.smithi103.stdout:Discarding blocks...Done. 2024-03-25T19:19:03.911 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi103.front.sepia.ceph.com -o noatime 2024-03-25T19:19:03.911 DEBUG:teuthology.orchestra.run.smithi103:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-25T19:19:04.039 DEBUG:teuthology.orchestra.run.smithi103:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-25T19:19:04.074 DEBUG:teuthology.orchestra.run.smithi103:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:19:04.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:04.180+0000 7f2ac24f9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-25T19:19:04.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:04.180+0000 7f2ac24f9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-25T19:19:04.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:04.180+0000 7f2ac24f9640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:19:04.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:04.180+0000 7f2ac24f9640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:19:04.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:04.180+0000 7f2ac24f9640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:19:04.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:04.180+0000 7f2ac24f9640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-25T19:19:07.031 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-25T19:19:07.062 DEBUG:teuthology.orchestra.run.smithi103:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:19:07.154 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:07.154+0000 7f0971921640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-25T19:19:07.154 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:07.155+0000 7f0971921640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-25T19:19:07.154 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:07.155+0000 7f0971921640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:19:07.154 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:07.155+0000 7f0971921640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:19:07.154 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:07.155+0000 7f0971921640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:19:07.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:07.155+0000 7f0971921640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-25T19:19:10.007 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-25T19:19:10.038 DEBUG:teuthology.orchestra.run.smithi103:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:19:10.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:10.128+0000 7fc1d2b1b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-25T19:19:10.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:10.129+0000 7fc1d2b1b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-25T19:19:10.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:10.129+0000 7fc1d2b1b640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:19:10.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:10.129+0000 7fc1d2b1b640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:19:10.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:10.129+0000 7fc1d2b1b640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:19:10.130 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:10.129+0000 7fc1d2b1b640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-25T19:19:12.977 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-25T19:19:13.008 DEBUG:teuthology.orchestra.run.smithi103:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:19:13.099 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:13.098+0000 7f62d8e0c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-25T19:19:13.099 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:13.098+0000 7f62d8e0c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-25T19:19:13.099 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:13.098+0000 7f62d8e0c640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:19:13.099 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:13.098+0000 7f62d8e0c640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:19:13.099 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:13.098+0000 7f62d8e0c640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:19:13.099 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:13.099+0000 7f62d8e0c640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-25T19:19:15.941 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-25T19:19:15.972 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-25T19:19:15.972 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-25T19:19:15.972 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-25T19:19:16.003 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-25T19:19:16.003 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-25T19:19:16.073 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-25T19:19:16.073 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-25T19:19:16.142 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-25T19:19:16.143 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-25T19:19:16.211 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-25T19:19:16.211 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-25T19:19:16.279 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-03-25T19:19:16.279 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-25T19:19:16.311 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-03-25T19:19:16.311 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-25T19:19:16.383 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-03-25T19:19:16.383 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-25T19:19:16.453 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-03-25T19:19:16.453 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-25T19:19:16.522 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-03-25T19:19:16.522 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-25T19:19:16.590 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-25T19:19:16.590 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-25T19:19:16.606 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-03-25T19:19:16.606 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-25T19:19:16.646 INFO:tasks.ceph:Adding keys to all mons... 2024-03-25T19:19:16.646 DEBUG:teuthology.orchestra.run.smithi083:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T19:19:16.649 DEBUG:teuthology.orchestra.run.smithi103:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T19:19:16.689 INFO:teuthology.orchestra.run.smithi083.stdout:[mgr.y] 2024-03-25T19:19:16.689 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQAVzgFmDt7aLBAAOQoPfd+FJ8DS+IIDegr0cA== 2024-03-25T19:19:16.690 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.0] 2024-03-25T19:19:16.690 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQAZzgFmrERyCRAABgikdYv63zGb2qhFQ9OO/A== 2024-03-25T19:19:16.690 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.1] 2024-03-25T19:19:16.690 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQAczgFmK1x0BxAAVC0mkuAslWsLmKZAORDA5A== 2024-03-25T19:19:16.690 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.2] 2024-03-25T19:19:16.690 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQAfzgFmb1j/BBAACOQFfFEO7FQn8+bkWe/Iiw== 2024-03-25T19:19:16.690 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.3] 2024-03-25T19:19:16.690 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQAizgFmxk9LAxAA1+t2sV4qfNHheg2qaybrWQ== 2024-03-25T19:19:16.690 INFO:teuthology.orchestra.run.smithi083.stdout:[mgr.x] 2024-03-25T19:19:16.690 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQAVzgFm3CytMRAAMAqVASYiPFCvSKUQd2KDwA== 2024-03-25T19:19:16.690 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.4] 2024-03-25T19:19:16.690 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQAozgFmstnDChAAQ2iH6levIwjPT/svlkLY1Q== 2024-03-25T19:19:16.690 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.5] 2024-03-25T19:19:16.690 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQArzgFmdRlDCRAAJ2+XaXJXPN98Z22SkrjB1Q== 2024-03-25T19:19:16.691 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.6] 2024-03-25T19:19:16.691 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQAuzgFm9ZzFBxAAbZvEdm0nu2VuycF55BwolQ== 2024-03-25T19:19:16.691 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.7] 2024-03-25T19:19:16.691 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQAxzgFmrvb0BRAASF/ZjrJjR4hbjq95BD/QNA== 2024-03-25T19:19:16.691 INFO:teuthology.orchestra.run.smithi083.stdout:[client.0] 2024-03-25T19:19:16.691 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQAVzgFmNshSNBAAPCr9hfM4SoaE03eZT4CPBw== 2024-03-25T19:19:16.691 INFO:teuthology.orchestra.run.smithi083.stdout:[client.1] 2024-03-25T19:19:16.691 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQAVzgFmX0i5NxAADxkfdWO+J3Ppf/se10mdWg== 2024-03-25T19:19:16.712 INFO:teuthology.orchestra.run.smithi103.stdout:[mgr.y] 2024-03-25T19:19:16.712 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQAVzgFmDt7aLBAAOQoPfd+FJ8DS+IIDegr0cA== 2024-03-25T19:19:16.712 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.0] 2024-03-25T19:19:16.712 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQAZzgFmrERyCRAABgikdYv63zGb2qhFQ9OO/A== 2024-03-25T19:19:16.712 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.1] 2024-03-25T19:19:16.712 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQAczgFmK1x0BxAAVC0mkuAslWsLmKZAORDA5A== 2024-03-25T19:19:16.712 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.2] 2024-03-25T19:19:16.712 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQAfzgFmb1j/BBAACOQFfFEO7FQn8+bkWe/Iiw== 2024-03-25T19:19:16.712 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.3] 2024-03-25T19:19:16.713 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQAizgFmxk9LAxAA1+t2sV4qfNHheg2qaybrWQ== 2024-03-25T19:19:16.713 INFO:teuthology.orchestra.run.smithi103.stdout:[mgr.x] 2024-03-25T19:19:16.713 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQAVzgFm3CytMRAAMAqVASYiPFCvSKUQd2KDwA== 2024-03-25T19:19:16.713 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.4] 2024-03-25T19:19:16.713 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQAozgFmstnDChAAQ2iH6levIwjPT/svlkLY1Q== 2024-03-25T19:19:16.713 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.5] 2024-03-25T19:19:16.713 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQArzgFmdRlDCRAAJ2+XaXJXPN98Z22SkrjB1Q== 2024-03-25T19:19:16.713 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.6] 2024-03-25T19:19:16.713 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQAuzgFm9ZzFBxAAbZvEdm0nu2VuycF55BwolQ== 2024-03-25T19:19:16.713 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.7] 2024-03-25T19:19:16.713 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQAxzgFmrvb0BRAASF/ZjrJjR4hbjq95BD/QNA== 2024-03-25T19:19:16.713 INFO:teuthology.orchestra.run.smithi103.stdout:[client.0] 2024-03-25T19:19:16.713 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQAVzgFmNshSNBAAPCr9hfM4SoaE03eZT4CPBw== 2024-03-25T19:19:16.713 INFO:teuthology.orchestra.run.smithi103.stdout:[client.1] 2024-03-25T19:19:16.713 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQAVzgFmX0i5NxAADxkfdWO+J3Ppf/se10mdWg== 2024-03-25T19:19:16.714 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:19:16.735 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:19:16.807 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:19:16.810 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:19:16.862 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:19:16.864 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:19:16.916 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:19:16.918 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:19:16.969 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:19:16.971 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:19:17.024 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:19:17.026 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:19:17.078 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:19:17.080 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:19:17.131 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:19:17.134 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:19:17.184 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:19:17.186 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:19:17.238 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:19:17.240 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:19:17.292 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:19:17.294 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:19:17.349 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:19:17.351 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:19:17.406 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-25T19:19:17.406 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-25T19:19:17.432 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:19:17.528 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-25T19:19:17.554 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-25T19:19:17.621 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:19:17.718 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-25T19:19:17.744 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-25T19:19:17.771 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:19:17.867 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-25T19:19:17.893 DEBUG:teuthology.orchestra.run.smithi083:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:19:17.895 DEBUG:teuthology.orchestra.run.smithi103:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:19:17.948 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-25T19:19:17.948 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:19:17.948 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-25T19:19:17.948 DEBUG:teuthology.orchestra.run.smithi083:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-25T19:19:17.951 INFO:tasks.ceph.mon.a:Started 2024-03-25T19:19:17.951 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:19:17.951 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-25T19:19:17.951 DEBUG:teuthology.orchestra.run.smithi083:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-25T19:19:17.953 INFO:tasks.ceph.mon.c:Started 2024-03-25T19:19:17.953 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:19:17.953 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-25T19:19:17.953 DEBUG:teuthology.orchestra.run.smithi103:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-25T19:19:17.990 INFO:tasks.ceph.mon.b:Started 2024-03-25T19:19:17.990 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-25T19:19:17.991 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-25T19:19:17.991 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-25T19:19:17.993 INFO:tasks.ceph.mgr.y:Started 2024-03-25T19:19:17.993 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-25T19:19:17.993 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-25T19:19:17.995 INFO:tasks.ceph.mgr.x:Started 2024-03-25T19:19:17.995 INFO:tasks.ceph:Setting crush tunables to default 2024-03-25T19:19:17.995 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --cluster ceph osd crush tunables default 2024-03-25T19:19:18.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:18.097+0000 7fd393c63640 10 monclient: get_monmap_and_config 2024-03-25T19:19:18.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:18.097+0000 7fd393c63640 10 monclient: build_initial_monmap 2024-03-25T19:19:18.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:18.097+0000 7fd393c63640 10 monclient: monmap: 2024-03-25T19:19:18.097 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T19:19:18.097 INFO:teuthology.orchestra.run.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:18.097 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T19:19:18.097993+0000 2024-03-25T19:19:18.097 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T19:19:18.097993+0000 2024-03-25T19:19:18.097 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:19:18.097 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 3 2024-03-25T19:19:18.097 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:19:18.097 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:19:18.097 INFO:teuthology.orchestra.run.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:19:18.097 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T19:19:18.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:18.097+0000 7fd393c63640 1 Processor -- start 2024-03-25T19:19:18.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:18.098+0000 7fd393c63640 1 -- start start 2024-03-25T19:19:18.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:18.098+0000 7fd393c63640 10 monclient: init 2024-03-25T19:19:18.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:18.098+0000 7fd393c63640 10 monclient: _reopen_session rank -1 2024-03-25T19:19:18.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:18.098+0000 7fd393c63640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:19:18.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:18.098+0000 7fd393c63640 10 monclient(hunting): picked mon.noname-b con 0x7fd38c0a9920 addr v1:172.21.15.83:6790/0 2024-03-25T19:19:18.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:18.098+0000 7fd393c63640 10 monclient(hunting): picked mon.noname-c con 0x7fd38c0b45a0 addr v1:172.21.15.103:6789/0 2024-03-25T19:19:18.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:18.098+0000 7fd393c63640 10 monclient(hunting): picked mon.noname-a con 0x7fd38c146450 addr v1:172.21.15.83:6789/0 2024-03-25T19:19:18.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:18.098+0000 7fd393c63640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd38c06b100 con 0x7fd38c146450 2024-03-25T19:19:18.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:18.098+0000 7fd393c63640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd38c0b2940 con 0x7fd38c0b45a0 2024-03-25T19:19:18.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:18.098+0000 7fd393c63640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd38c13fec0 con 0x7fd38c0a9920 2024-03-25T19:19:18.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:18.098+0000 7fd393c63640 10 monclient(hunting): _renew_subs 2024-03-25T19:19:18.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:18.098+0000 7fd393c63640 10 monclient(hunting): authenticate will time out at 6015.370461s 2024-03-25T19:19:18.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:18.098+0000 7fd392200640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fd38c146450 legacy=0x7fd38c14dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-25T19:19:18.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:18.098+0000 7fd3919ff640 1 -- >> v1:172.21.15.83:6790/0 conn(0x7fd38c0a9920 legacy=0x7fd38c13f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6790/0 2024-03-25T19:19:18.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:18.098+0000 7fd3911fe640 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fd38c0b45a0 legacy=0x7fd38c145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2024-03-25T19:19:18.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:18.299+0000 7fd3919ff640 1 -- >> v1:172.21.15.83:6790/0 conn(0x7fd38c0a9920 legacy=0x7fd38c13f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6790/0 2024-03-25T19:19:18.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:18.299+0000 7fd392200640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fd38c146450 legacy=0x7fd38c14dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-25T19:19:18.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:18.299+0000 7fd3911fe640 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fd38c0b45a0 legacy=0x7fd38c145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2024-03-25T19:19:18.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:18.700+0000 7fd3919ff640 1 -- >> v1:172.21.15.83:6790/0 conn(0x7fd38c0a9920 legacy=0x7fd38c13f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6790/0 2024-03-25T19:19:18.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:18.700+0000 7fd392200640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fd38c146450 legacy=0x7fd38c14dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-25T19:19:18.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:18.700+0000 7fd3911fe640 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fd38c0b45a0 legacy=0x7fd38c145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2024-03-25T19:19:19.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:19.501+0000 7fd3919ff640 1 -- >> v1:172.21.15.83:6790/0 conn(0x7fd38c0a9920 legacy=0x7fd38c13f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6790/0 2024-03-25T19:19:19.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:19.501+0000 7fd392200640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fd38c146450 legacy=0x7fd38c14dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-25T19:19:19.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:19.501+0000 7fd3911fe640 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fd38c0b45a0 legacy=0x7fd38c145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2024-03-25T19:19:21.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:21.098+0000 7fd3837fe640 10 monclient(hunting): tick 2024-03-25T19:19:21.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:21.098+0000 7fd3837fe640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:19:21.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:21.098+0000 7fd3837fe640 1 monclient(hunting): continuing hunt 2024-03-25T19:19:21.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:21.098+0000 7fd3837fe640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:19:21.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:21.098+0000 7fd3837fe640 1 -- >> v1:172.21.15.83:6790/0 conn(0x7fd38c0a9920 legacy=0x7fd38c13f7b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:19:21.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:21.098+0000 7fd3837fe640 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fd38c0b45a0 legacy=0x7fd38c145d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:19:21.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:21.098+0000 7fd3837fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fd38c146450 legacy=0x7fd38c14dc70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:19:21.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:21.098+0000 7fd3837fe640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:19:21.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:21.098+0000 7fd3837fe640 10 monclient(hunting): picked mon.noname-a con 0x7fd370003680 addr v1:172.21.15.83:6789/0 2024-03-25T19:19:21.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:21.098+0000 7fd3837fe640 10 monclient(hunting): picked mon.noname-b con 0x7fd38c0a9920 addr v1:172.21.15.83:6790/0 2024-03-25T19:19:21.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:21.098+0000 7fd3837fe640 10 monclient(hunting): picked mon.noname-c con 0x7fd38c0b45a0 addr v1:172.21.15.103:6789/0 2024-03-25T19:19:21.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:21.098+0000 7fd3837fe640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd38c06b100 con 0x7fd370003680 2024-03-25T19:19:21.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:21.098+0000 7fd3837fe640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd38c0b2940 con 0x7fd38c0b45a0 2024-03-25T19:19:21.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:21.098+0000 7fd3837fe640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd38c13fec0 con 0x7fd38c0a9920 2024-03-25T19:19:21.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:21.098+0000 7fd3837fe640 10 monclient(hunting): _renew_subs 2024-03-25T19:19:21.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:21.098+0000 7fd3919ff640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fd370003680 legacy=0x7fd370005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-25T19:19:21.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:21.098+0000 7fd3911fe640 1 -- >> v1:172.21.15.83:6790/0 conn(0x7fd38c0a9920 legacy=0x7fd3700093a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6790/0 2024-03-25T19:19:21.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:21.098+0000 7fd392200640 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fd38c0b45a0 legacy=0x7fd37000cba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2024-03-25T19:19:21.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:21.299+0000 7fd3919ff640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fd370003680 legacy=0x7fd370005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-25T19:19:21.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:21.299+0000 7fd3911fe640 1 -- >> v1:172.21.15.83:6790/0 conn(0x7fd38c0a9920 legacy=0x7fd3700093a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6790/0 2024-03-25T19:19:21.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:21.299+0000 7fd392200640 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fd38c0b45a0 legacy=0x7fd37000cba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2024-03-25T19:19:21.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:21.700+0000 7fd3919ff640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fd370003680 legacy=0x7fd370005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-25T19:19:21.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:21.700+0000 7fd3911fe640 1 -- >> v1:172.21.15.83:6790/0 conn(0x7fd38c0a9920 legacy=0x7fd3700093a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6790/0 2024-03-25T19:19:21.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:21.700+0000 7fd392200640 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fd38c0b45a0 legacy=0x7fd37000cba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2024-03-25T19:19:22.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:22.501+0000 7fd3919ff640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fd370003680 legacy=0x7fd370005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-25T19:19:22.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:22.501+0000 7fd3911fe640 1 -- >> v1:172.21.15.83:6790/0 conn(0x7fd38c0a9920 legacy=0x7fd3700093a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6790/0 2024-03-25T19:19:22.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:22.501+0000 7fd392200640 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fd38c0b45a0 legacy=0x7fd37000cba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2024-03-25T19:19:24.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:24.098+0000 7fd3837fe640 10 monclient(hunting): tick 2024-03-25T19:19:24.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:24.098+0000 7fd3837fe640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:19:24.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:24.098+0000 7fd3837fe640 1 monclient(hunting): continuing hunt 2024-03-25T19:19:24.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:24.098+0000 7fd3837fe640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:19:24.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:24.098+0000 7fd3837fe640 1 -- >> v1:172.21.15.83:6790/0 conn(0x7fd38c0a9920 legacy=0x7fd3700093a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:19:24.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:24.098+0000 7fd3837fe640 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fd38c0b45a0 legacy=0x7fd37000cba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:19:24.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:24.098+0000 7fd3837fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fd370003680 legacy=0x7fd370005ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:19:24.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:24.099+0000 7fd3837fe640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:19:24.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:24.099+0000 7fd3837fe640 10 monclient(hunting): picked mon.noname-b con 0x7fd38c0a9920 addr v1:172.21.15.83:6790/0 2024-03-25T19:19:24.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:24.099+0000 7fd3837fe640 10 monclient(hunting): picked mon.noname-a con 0x7fd370003680 addr v1:172.21.15.83:6789/0 2024-03-25T19:19:24.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:24.099+0000 7fd3837fe640 10 monclient(hunting): picked mon.noname-c con 0x7fd38c0b45a0 addr v1:172.21.15.103:6789/0 2024-03-25T19:19:24.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:24.099+0000 7fd3837fe640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd370011350 con 0x7fd370003680 2024-03-25T19:19:24.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:24.099+0000 7fd3837fe640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd38c06b100 con 0x7fd38c0b45a0 2024-03-25T19:19:24.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:24.099+0000 7fd3837fe640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd38c0b2940 con 0x7fd38c0a9920 2024-03-25T19:19:24.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:24.099+0000 7fd3837fe640 10 monclient(hunting): _renew_subs 2024-03-25T19:19:24.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:24.099+0000 7fd3919ff640 1 -- >> v1:172.21.15.83:6790/0 conn(0x7fd38c0a9920 legacy=0x7fd370009ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6790/0 2024-03-25T19:19:24.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:24.099+0000 7fd3911fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fd370003680 legacy=0x7fd3700093a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-25T19:19:24.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:24.099+0000 7fd392200640 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fd38c0b45a0 legacy=0x7fd37000a1e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2024-03-25T19:19:24.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:24.300+0000 7fd3919ff640 1 -- >> v1:172.21.15.83:6790/0 conn(0x7fd38c0a9920 legacy=0x7fd370009ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6790/0 2024-03-25T19:19:24.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:24.300+0000 7fd3911fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fd370003680 legacy=0x7fd3700093a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-25T19:19:24.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:24.300+0000 7fd392200640 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fd38c0b45a0 legacy=0x7fd37000a1e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2024-03-25T19:19:24.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:24.701+0000 7fd3919ff640 1 -- >> v1:172.21.15.83:6790/0 conn(0x7fd38c0a9920 legacy=0x7fd370009ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6790/0 2024-03-25T19:19:24.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:24.701+0000 7fd3911fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fd370003680 legacy=0x7fd3700093a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-25T19:19:24.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:24.701+0000 7fd392200640 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fd38c0b45a0 legacy=0x7fd37000a1e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2024-03-25T19:19:25.002 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:00:06.912 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:19:25.080 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:00:07.008 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:19:25.081 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:00:07.008 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:19:25.313 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:00:07.223 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:19:25.401 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:00:07.328 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:19:25.401 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:00:07.328 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:19:25.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:25.502+0000 7fd3919ff640 1 -- >> v1:172.21.15.83:6790/0 conn(0x7fd38c0a9920 legacy=0x7fd370009ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6790/0 2024-03-25T19:19:25.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:25.502+0000 7fd3911fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fd370003680 legacy=0x7fd3700093a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-25T19:19:25.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:25.502+0000 7fd392200640 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fd38c0b45a0 legacy=0x7fd37000a1e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.103:6789/0 2024-03-25T19:19:27.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:27.099+0000 7fd3837fe640 10 monclient(hunting): tick 2024-03-25T19:19:27.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:27.099+0000 7fd3837fe640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:19:27.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:27.099+0000 7fd3837fe640 1 monclient(hunting): continuing hunt 2024-03-25T19:19:27.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:27.099+0000 7fd3837fe640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:19:27.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:27.099+0000 7fd3837fe640 1 -- >> v1:172.21.15.83:6790/0 conn(0x7fd38c0a9920 legacy=0x7fd370009ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:19:27.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:27.099+0000 7fd3837fe640 1 -- >> v1:172.21.15.103:6789/0 conn(0x7fd38c0b45a0 legacy=0x7fd37000a1e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:19:27.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:27.099+0000 7fd3837fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fd370003680 legacy=0x7fd3700093a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:19:27.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:27.099+0000 7fd3837fe640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:19:27.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:27.099+0000 7fd3837fe640 10 monclient(hunting): picked mon.noname-c con 0x7fd38c0b45a0 addr v1:172.21.15.103:6789/0 2024-03-25T19:19:27.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:27.099+0000 7fd3837fe640 10 monclient(hunting): picked mon.noname-a con 0x7fd370003680 addr v1:172.21.15.83:6789/0 2024-03-25T19:19:27.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:27.099+0000 7fd3837fe640 10 monclient(hunting): picked mon.noname-b con 0x7fd38c0a9920 addr v1:172.21.15.83:6790/0 2024-03-25T19:19:27.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:27.099+0000 7fd3837fe640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd38c13fec0 con 0x7fd370003680 2024-03-25T19:19:27.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:27.099+0000 7fd3837fe640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd370011350 con 0x7fd38c0b45a0 2024-03-25T19:19:27.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:27.099+0000 7fd3837fe640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd38c06b100 con 0x7fd38c0a9920 2024-03-25T19:19:27.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:27.099+0000 7fd3837fe640 10 monclient(hunting): _renew_subs 2024-03-25T19:19:27.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:27.138+0000 7fd3919ff640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fd38c0b45a0 0x7fd37000a1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:39658/0 (socket says 172.21.15.83:39658) 2024-03-25T19:19:27.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:27.138+0000 7fd3919ff640 1 -- 172.21.15.83:0/2943942012 learned_addr learned my addr 172.21.15.83:0/2943942012 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:19:30.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:30.098+0000 7fd3837fe640 10 monclient(hunting): tick 2024-03-25T19:19:30.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:30.098+0000 7fd3837fe640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:19:30.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:30.098+0000 7fd3837fe640 1 monclient(hunting): continuing hunt 2024-03-25T19:19:30.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:30.098+0000 7fd3837fe640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:19:30.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:30.098+0000 7fd3837fe640 1 -- 172.21.15.83:0/2943942012 >> v1:172.21.15.83:6790/0 conn(0x7fd38c0a9920 legacy=0x7fd370009ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:30.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:30.098+0000 7fd3837fe640 1 -- 172.21.15.83:0/2943942012 >> v1:172.21.15.103:6789/0 conn(0x7fd38c0b45a0 legacy=0x7fd37000a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:30.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:30.098+0000 7fd3837fe640 1 -- 172.21.15.83:0/2943942012 >> v1:172.21.15.83:6789/0 conn(0x7fd370003680 legacy=0x7fd370005a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:30.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:30.098+0000 7fd3837fe640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:19:30.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:30.098+0000 7fd3837fe640 10 monclient(hunting): picked mon.noname-b con 0x7fd37000fc20 addr v1:172.21.15.83:6790/0 2024-03-25T19:19:30.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:30.098+0000 7fd3837fe640 10 monclient(hunting): picked mon.noname-c con 0x7fd38c0b45a0 addr v1:172.21.15.103:6789/0 2024-03-25T19:19:30.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:30.098+0000 7fd3837fe640 10 monclient(hunting): picked mon.noname-a con 0x7fd370009ad0 addr v1:172.21.15.83:6789/0 2024-03-25T19:19:30.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:30.098+0000 7fd3837fe640 1 -- 172.21.15.83:0/2943942012 --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd38c0b2940 con 0x7fd370009ad0 2024-03-25T19:19:30.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:30.098+0000 7fd3837fe640 1 -- 172.21.15.83:0/2943942012 --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd370009eb0 con 0x7fd38c0b45a0 2024-03-25T19:19:30.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:30.098+0000 7fd3837fe640 1 -- 172.21.15.83:0/2943942012 --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd370005a70 con 0x7fd37000fc20 2024-03-25T19:19:30.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:30.098+0000 7fd3837fe640 10 monclient(hunting): _renew_subs 2024-03-25T19:19:33.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:33.098+0000 7fd3837fe640 10 monclient(hunting): tick 2024-03-25T19:19:33.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:33.098+0000 7fd3837fe640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:19:33.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:33.098+0000 7fd3837fe640 1 monclient(hunting): continuing hunt 2024-03-25T19:19:33.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:33.098+0000 7fd3837fe640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:19:33.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:33.098+0000 7fd3837fe640 1 -- 172.21.15.83:0/2943942012 >> v1:172.21.15.83:6790/0 conn(0x7fd37000fc20 legacy=0x7fd3700093a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:33.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:33.098+0000 7fd3837fe640 1 -- 172.21.15.83:0/2943942012 >> v1:172.21.15.103:6789/0 conn(0x7fd38c0b45a0 legacy=0x7fd37000a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:33.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:33.098+0000 7fd3837fe640 1 -- 172.21.15.83:0/2943942012 >> v1:172.21.15.83:6789/0 conn(0x7fd370009ad0 legacy=0x7fd37000cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:33.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:33.098+0000 7fd3837fe640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:19:33.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:33.098+0000 7fd3837fe640 10 monclient(hunting): picked mon.noname-b con 0x7fd37000d310 addr v1:172.21.15.83:6790/0 2024-03-25T19:19:33.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:33.098+0000 7fd3837fe640 10 monclient(hunting): picked mon.noname-a con 0x7fd37000f760 addr v1:172.21.15.83:6789/0 2024-03-25T19:19:33.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:33.098+0000 7fd3837fe640 10 monclient(hunting): picked mon.noname-c con 0x7fd38c0b45a0 addr v1:172.21.15.103:6789/0 2024-03-25T19:19:33.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:33.098+0000 7fd3837fe640 1 -- 172.21.15.83:0/2943942012 --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd370005c00 con 0x7fd37000f760 2024-03-25T19:19:33.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:33.098+0000 7fd3837fe640 1 -- 172.21.15.83:0/2943942012 --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd370005fd0 con 0x7fd38c0b45a0 2024-03-25T19:19:33.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:33.098+0000 7fd3837fe640 1 -- 172.21.15.83:0/2943942012 --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd370009440 con 0x7fd37000d310 2024-03-25T19:19:33.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:33.098+0000 7fd3837fe640 10 monclient(hunting): _renew_subs 2024-03-25T19:19:36.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:36.100+0000 7fd3837fe640 10 monclient(hunting): tick 2024-03-25T19:19:36.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:36.100+0000 7fd3837fe640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:19:36.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:36.100+0000 7fd3837fe640 1 monclient(hunting): continuing hunt 2024-03-25T19:19:36.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:36.100+0000 7fd3837fe640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:19:36.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:36.100+0000 7fd3837fe640 1 -- 172.21.15.83:0/2943942012 >> v1:172.21.15.83:6790/0 conn(0x7fd37000d310 legacy=0x7fd3700125b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:36.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:36.100+0000 7fd3837fe640 1 -- 172.21.15.83:0/2943942012 >> v1:172.21.15.103:6789/0 conn(0x7fd38c0b45a0 legacy=0x7fd37000a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:36.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:36.100+0000 7fd3837fe640 1 -- 172.21.15.83:0/2943942012 >> v1:172.21.15.83:6789/0 conn(0x7fd37000f760 legacy=0x7fd3700115a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:36.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:36.100+0000 7fd3837fe640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:19:36.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:36.100+0000 7fd3837fe640 10 monclient(hunting): picked mon.noname-a con 0x7fd370011cb0 addr v1:172.21.15.83:6789/0 2024-03-25T19:19:36.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:36.100+0000 7fd3837fe640 10 monclient(hunting): picked mon.noname-b con 0x7fd37000d310 addr v1:172.21.15.83:6790/0 2024-03-25T19:19:36.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:36.100+0000 7fd3837fe640 10 monclient(hunting): picked mon.noname-c con 0x7fd38c0b45a0 addr v1:172.21.15.103:6789/0 2024-03-25T19:19:36.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:36.100+0000 7fd3837fe640 1 -- 172.21.15.83:0/2943942012 --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd370009670 con 0x7fd370011cb0 2024-03-25T19:19:36.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:36.100+0000 7fd3837fe640 1 -- 172.21.15.83:0/2943942012 --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3700115a0 con 0x7fd38c0b45a0 2024-03-25T19:19:36.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:36.100+0000 7fd3837fe640 1 -- 172.21.15.83:0/2943942012 --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd370011730 con 0x7fd37000d310 2024-03-25T19:19:36.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:36.100+0000 7fd3837fe640 10 monclient(hunting): _renew_subs 2024-03-25T19:19:39.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:39.100+0000 7fd3837fe640 10 monclient(hunting): tick 2024-03-25T19:19:39.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:39.100+0000 7fd3837fe640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:19:39.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:39.100+0000 7fd3837fe640 1 monclient(hunting): continuing hunt 2024-03-25T19:19:39.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:39.100+0000 7fd3837fe640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:19:39.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:39.100+0000 7fd3837fe640 1 -- 172.21.15.83:0/2943942012 >> v1:172.21.15.83:6790/0 conn(0x7fd37000d310 legacy=0x7fd370012090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:39.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:39.100+0000 7fd3837fe640 1 -- 172.21.15.83:0/2943942012 >> v1:172.21.15.103:6789/0 conn(0x7fd38c0b45a0 legacy=0x7fd37000a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:39.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:39.100+0000 7fd3837fe640 1 -- 172.21.15.83:0/2943942012 >> v1:172.21.15.83:6789/0 conn(0x7fd370011cb0 legacy=0x7fd37000cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:39.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:39.100+0000 7fd3837fe640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:19:39.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:39.100+0000 7fd3837fe640 10 monclient(hunting): picked mon.noname-b con 0x7fd370007390 addr v1:172.21.15.83:6790/0 2024-03-25T19:19:39.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:39.100+0000 7fd3837fe640 10 monclient(hunting): picked mon.noname-c con 0x7fd370008ed0 addr v1:172.21.15.103:6789/0 2024-03-25T19:19:39.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:39.100+0000 7fd3837fe640 10 monclient(hunting): picked mon.noname-a con 0x7fd370011cb0 addr v1:172.21.15.83:6789/0 2024-03-25T19:19:39.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:39.100+0000 7fd3837fe640 1 -- 172.21.15.83:0/2943942012 --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd370011960 con 0x7fd370011cb0 2024-03-25T19:19:39.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:39.100+0000 7fd3837fe640 1 -- 172.21.15.83:0/2943942012 --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd37000a1e0 con 0x7fd370008ed0 2024-03-25T19:19:39.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:39.100+0000 7fd3837fe640 1 -- 172.21.15.83:0/2943942012 --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd37000a3c0 con 0x7fd370007390 2024-03-25T19:19:39.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:39.100+0000 7fd3837fe640 10 monclient(hunting): _renew_subs 2024-03-25T19:19:42.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:42.100+0000 7fd3837fe640 10 monclient(hunting): tick 2024-03-25T19:19:42.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:42.100+0000 7fd3837fe640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:19:42.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:42.100+0000 7fd3837fe640 1 monclient(hunting): continuing hunt 2024-03-25T19:19:42.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:42.100+0000 7fd3837fe640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:19:42.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:42.100+0000 7fd3837fe640 1 -- 172.21.15.83:0/2943942012 >> v1:172.21.15.83:6790/0 conn(0x7fd370007390 legacy=0x7fd3700077b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:42.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:42.100+0000 7fd3837fe640 1 -- 172.21.15.83:0/2943942012 >> v1:172.21.15.103:6789/0 conn(0x7fd370008ed0 legacy=0x7fd3700177d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:42.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:42.100+0000 7fd3837fe640 1 -- 172.21.15.83:0/2943942012 >> v1:172.21.15.83:6789/0 conn(0x7fd370011cb0 legacy=0x7fd370010300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:42.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:42.100+0000 7fd3837fe640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:19:42.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:42.100+0000 7fd3837fe640 10 monclient(hunting): picked mon.noname-b con 0x7fd3700130a0 addr v1:172.21.15.83:6790/0 2024-03-25T19:19:42.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:42.100+0000 7fd3837fe640 10 monclient(hunting): picked mon.noname-a con 0x7fd370007770 addr v1:172.21.15.83:6789/0 2024-03-25T19:19:42.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:42.100+0000 7fd3837fe640 10 monclient(hunting): picked mon.noname-c con 0x7fd370008ed0 addr v1:172.21.15.103:6789/0 2024-03-25T19:19:42.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:42.100+0000 7fd3837fe640 1 -- 172.21.15.83:0/2943942012 --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd37000a5f0 con 0x7fd370007770 2024-03-25T19:19:42.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:42.100+0000 7fd3837fe640 1 -- 172.21.15.83:0/2943942012 --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd370011170 con 0x7fd370008ed0 2024-03-25T19:19:42.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:42.100+0000 7fd3837fe640 1 -- 172.21.15.83:0/2943942012 --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd37000cc90 con 0x7fd3700130a0 2024-03-25T19:19:42.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:42.100+0000 7fd3837fe640 10 monclient(hunting): _renew_subs 2024-03-25T19:19:45.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:45.100+0000 7fd3837fe640 10 monclient(hunting): tick 2024-03-25T19:19:45.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:45.100+0000 7fd3837fe640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:19:45.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:45.100+0000 7fd3837fe640 1 monclient(hunting): continuing hunt 2024-03-25T19:19:45.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:45.100+0000 7fd3837fe640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:19:45.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:45.100+0000 7fd3837fe640 1 -- 172.21.15.83:0/2943942012 >> v1:172.21.15.83:6790/0 conn(0x7fd3700130a0 legacy=0x7fd370010a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:45.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:45.101+0000 7fd3837fe640 1 -- 172.21.15.83:0/2943942012 >> v1:172.21.15.103:6789/0 conn(0x7fd370008ed0 legacy=0x7fd370010300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:45.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:45.101+0000 7fd3837fe640 1 -- 172.21.15.83:0/2943942012 >> v1:172.21.15.83:6789/0 conn(0x7fd370007770 legacy=0x7fd370007b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:45.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:45.101+0000 7fd3837fe640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:19:45.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:45.101+0000 7fd3837fe640 10 monclient(hunting): picked mon.noname-c con 0x7fd370008ed0 addr v1:172.21.15.103:6789/0 2024-03-25T19:19:45.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:45.101+0000 7fd3837fe640 10 monclient(hunting): picked mon.noname-b con 0x7fd3700130a0 addr v1:172.21.15.83:6790/0 2024-03-25T19:19:45.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:45.101+0000 7fd3837fe640 10 monclient(hunting): picked mon.noname-a con 0x7fd370007770 addr v1:172.21.15.83:6789/0 2024-03-25T19:19:45.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:45.101+0000 7fd3837fe640 1 -- 172.21.15.83:0/2943942012 --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd37000cec0 con 0x7fd370007770 2024-03-25T19:19:45.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:45.101+0000 7fd3837fe640 1 -- 172.21.15.83:0/2943942012 --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd37000d050 con 0x7fd370008ed0 2024-03-25T19:19:45.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:45.101+0000 7fd3837fe640 1 -- 172.21.15.83:0/2943942012 --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd370008b60 con 0x7fd3700130a0 2024-03-25T19:19:45.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:45.101+0000 7fd3837fe640 10 monclient(hunting): _renew_subs 2024-03-25T19:19:46.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.094+0000 7fd3909fd640 1 -- 172.21.15.83:0/2943942012 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 715407873 0 0) 0x7fd37000cec0 con 0x7fd370007770 2024-03-25T19:19:46.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.094+0000 7fd3909fd640 10 monclient(hunting): _negotiate 2024-03-25T19:19:46.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.094+0000 7fd3909fd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:46.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.094+0000 7fd3909fd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:46.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.094+0000 7fd3909fd640 10 monclient(hunting): my global_id is 4101 2024-03-25T19:19:46.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.094+0000 7fd3909fd640 1 -- 172.21.15.83:0/2943942012 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd378003520 con 0x7fd370007770 2024-03-25T19:19:46.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.098+0000 7fd3909fd640 1 -- 172.21.15.83:0/2943942012 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1153642410 0 0) 0x7fd37000d050 con 0x7fd370008ed0 2024-03-25T19:19:46.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.098+0000 7fd3909fd640 10 monclient(hunting): _negotiate 2024-03-25T19:19:46.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.098+0000 7fd3909fd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:46.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.098+0000 7fd3909fd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:46.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.098+0000 7fd3909fd640 10 monclient(hunting): my global_id is 4105 2024-03-25T19:19:46.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.098+0000 7fd3909fd640 1 -- 172.21.15.83:0/2943942012 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd37000cec0 con 0x7fd370008ed0 2024-03-25T19:19:46.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.122+0000 7fd3909fd640 1 -- 172.21.15.83:0/2943942012 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 628783608 0 0) 0x7fd370008b60 con 0x7fd3700130a0 2024-03-25T19:19:46.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.122+0000 7fd3909fd640 10 monclient(hunting): _negotiate 2024-03-25T19:19:46.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.122+0000 7fd3909fd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:46.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.122+0000 7fd3909fd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:46.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.122+0000 7fd3909fd640 10 monclient(hunting): my global_id is 4106 2024-03-25T19:19:46.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.122+0000 7fd3909fd640 1 -- 172.21.15.83:0/2943942012 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd37000d050 con 0x7fd3700130a0 2024-03-25T19:19:46.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.177+0000 7fd3909fd640 1 -- 172.21.15.83:0/2943942012 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1252081353 0 0) 0x7fd378003520 con 0x7fd370007770 2024-03-25T19:19:46.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.177+0000 7fd3909fd640 1 -- 172.21.15.83:0/2943942012 >> v1:172.21.15.83:6790/0 conn(0x7fd3700130a0 legacy=0x7fd3700109f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:46.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.177+0000 7fd3909fd640 1 -- 172.21.15.83:0/2943942012 >> v1:172.21.15.103:6789/0 conn(0x7fd370008ed0 legacy=0x7fd3700102e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:46.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.177+0000 7fd3909fd640 10 monclient: _finish_hunting 0 2024-03-25T19:19:46.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.177+0000 7fd3909fd640 1 monclient: found mon.noname-a 2024-03-25T19:19:46.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.177+0000 7fd3909fd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:19:46.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.177+0000 7fd3909fd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.83:6789/0 2024-03-25T19:19:46.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.177+0000 7fd3909fd640 1 -- 172.21.15.83:0/2943942012 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd370008cf0 con 0x7fd370007770 2024-03-25T19:19:46.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.177+0000 7fd3909fd640 10 monclient: _finish_auth 0 2024-03-25T19:19:46.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.177+0000 7fd3909fd640 10 monclient: _check_auth_tickets 2024-03-25T19:19:46.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.177+0000 7fd3909fd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:19:46.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.177+0000 7fd393c63640 5 monclient: authenticate success, global_id 4101 2024-03-25T19:19:46.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.177+0000 7fd393c63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:19:46.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.181+0000 7fd3909fd640 1 -- 172.21.15.83:0/2943942012 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fd384003f40 con 0x7fd370007770 2024-03-25T19:19:46.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.181+0000 7fd3909fd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:46.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.181+0000 7fd3909fd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:19:46.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.181+0000 7fd3909fd640 10 monclient: dump: 2024-03-25T19:19:46.181 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T19:19:46.181 INFO:teuthology.orchestra.run.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:46.181 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:46.181 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:46.181 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:46.182 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 3 2024-03-25T19:19:46.182 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:46.182 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:46.182 INFO:teuthology.orchestra.run.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:46.182 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T19:19:46.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.181+0000 7fd393c63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:19:46.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.228+0000 7fd3909fd640 1 -- 172.21.15.83:0/2943942012 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd384004250 con 0x7fd370007770 2024-03-25T19:19:46.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.228+0000 7fd3909fd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:19:46.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.228+0000 7fd3909fd640 1 -- 172.21.15.83:0/2943942012 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fd3840045e0 con 0x7fd370007770 2024-03-25T19:19:46.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.228+0000 7fd3909fd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:46.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.228+0000 7fd3909fd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:19:46.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.228+0000 7fd3909fd640 10 monclient: dump: 2024-03-25T19:19:46.229 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T19:19:46.229 INFO:teuthology.orchestra.run.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:46.229 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:46.229 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:46.229 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:46.229 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 3 2024-03-25T19:19:46.229 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:46.229 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:46.229 INFO:teuthology.orchestra.run.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:46.229 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T19:19:46.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.228+0000 7fd393c63640 10 monclient: get_monmap_and_config success 2024-03-25T19:19:46.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.228+0000 7fd393c63640 4 set_mon_vals no callback set 2024-03-25T19:19:46.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.229+0000 7fd393c63640 10 monclient: shutdown 2024-03-25T19:19:46.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.229+0000 7fd393c63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:19:46.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.229+0000 7fd393c63640 1 -- 172.21.15.83:0/2943942012 >> v1:172.21.15.83:6789/0 conn(0x7fd370007770 legacy=0x7fd37000abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:46.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.229+0000 7fd393c63640 1 -- 172.21.15.83:0/2943942012 shutdown_connections 2024-03-25T19:19:46.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.229+0000 7fd393c63640 1 -- 172.21.15.83:0/2943942012 >> 172.21.15.83:0/2943942012 conn(0x7fd38c0b7f40 msgr2=0x7fd38c0b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:19:46.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.229+0000 7fd393c63640 1 -- 172.21.15.83:0/2943942012 shutdown_connections 2024-03-25T19:19:46.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.229+0000 7fd393c63640 1 -- 172.21.15.83:0/2943942012 wait complete. 2024-03-25T19:19:46.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.231+0000 7fd393c63640 10 monclient: build_initial_monmap 2024-03-25T19:19:46.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.231+0000 7fd393c63640 10 monclient: monmap: 2024-03-25T19:19:46.231 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T19:19:46.232 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:19:46.232 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T19:19:46.232 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T19:19:46.232 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:19:46.232 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T19:19:46.232 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:19:46.232 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:19:46.232 INFO:teuthology.orchestra.run.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:19:46.232 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T19:19:46.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.231+0000 7fd393c63640 1 librados: starting msgr at 2024-03-25T19:19:46.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.231+0000 7fd393c63640 1 librados: starting objecter 2024-03-25T19:19:46.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.231+0000 7fd393c63640 1 Processor -- start 2024-03-25T19:19:46.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.231+0000 7fd393c63640 1 -- start start 2024-03-25T19:19:46.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.231+0000 7fd393c63640 1 librados: setting wanted keys 2024-03-25T19:19:46.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.231+0000 7fd393c63640 1 librados: calling monclient init 2024-03-25T19:19:46.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.231+0000 7fd393c63640 10 monclient: init 2024-03-25T19:19:46.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.232+0000 7fd393c63640 10 monclient: _reopen_session rank -1 2024-03-25T19:19:46.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.232+0000 7fd393c63640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:19:46.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.232+0000 7fd393c63640 10 monclient(hunting): picked mon.noname-c con 0x7fd370007770 addr v1:172.21.15.83:6790/0 2024-03-25T19:19:46.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.232+0000 7fd393c63640 10 monclient(hunting): picked mon.noname-b con 0x7fd370008ed0 addr v1:172.21.15.103:6789/0 2024-03-25T19:19:46.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.232+0000 7fd393c63640 10 monclient(hunting): picked mon.noname-a con 0x7fd3700130a0 addr v1:172.21.15.83:6789/0 2024-03-25T19:19:46.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.232+0000 7fd393c63640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd38c142120 con 0x7fd3700130a0 2024-03-25T19:19:46.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.232+0000 7fd393c63640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd38c143280 con 0x7fd370008ed0 2024-03-25T19:19:46.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.232+0000 7fd393c63640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd38c141a30 con 0x7fd370007770 2024-03-25T19:19:46.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.232+0000 7fd393c63640 10 monclient(hunting): _renew_subs 2024-03-25T19:19:46.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.232+0000 7fd393c63640 10 monclient(hunting): authenticate will time out at 6043.504352s 2024-03-25T19:19:46.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.236+0000 7fd3911fe640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fd370008ed0 0x7fd38c1d7730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:49262/0 (socket says 172.21.15.83:49262) 2024-03-25T19:19:46.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.236+0000 7fd3911fe640 1 -- 172.21.15.83:0/3799604360 learned_addr learned my addr 172.21.15.83:0/3799604360 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:19:46.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.237+0000 7fd3827fc640 10 client.?.objecter ms_handle_connect 0x7fd370008ed0 2024-03-25T19:19:46.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.237+0000 7fd3827fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:46.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.247+0000 7fd3827fc640 10 client.?.objecter ms_handle_connect 0x7fd370007770 2024-03-25T19:19:46.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.247+0000 7fd3827fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:46.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.256+0000 7fd3827fc640 10 client.?.objecter ms_handle_connect 0x7fd3700130a0 2024-03-25T19:19:46.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.256+0000 7fd3827fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:46.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.310+0000 7fd3827fc640 1 -- 172.21.15.83:0/3799604360 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2811971379 0 0) 0x7fd38c142120 con 0x7fd3700130a0 2024-03-25T19:19:46.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.311+0000 7fd3827fc640 10 monclient(hunting): _negotiate 2024-03-25T19:19:46.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.311+0000 7fd3827fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:46.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.311+0000 7fd3827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:46.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.311+0000 7fd3827fc640 10 monclient(hunting): my global_id is 4113 2024-03-25T19:19:46.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.311+0000 7fd3827fc640 1 -- 172.21.15.83:0/3799604360 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd37000a820 con 0x7fd3700130a0 2024-03-25T19:19:46.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.336+0000 7fd3827fc640 1 -- 172.21.15.83:0/3799604360 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1172270041 0 0) 0x7fd37000d050 con 0x7fd370008ed0 2024-03-25T19:19:46.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.336+0000 7fd3827fc640 10 monclient(hunting): _negotiate 2024-03-25T19:19:46.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.336+0000 7fd3827fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:46.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.336+0000 7fd3827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:46.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.336+0000 7fd3827fc640 10 monclient(hunting): my global_id is 4114 2024-03-25T19:19:46.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.336+0000 7fd3827fc640 1 -- 172.21.15.83:0/3799604360 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd38c142120 con 0x7fd370008ed0 2024-03-25T19:19:46.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.360+0000 7fd3827fc640 1 -- 172.21.15.83:0/3799604360 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4253433440 0 0) 0x7fd37000a820 con 0x7fd3700130a0 2024-03-25T19:19:46.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.360+0000 7fd3827fc640 1 -- 172.21.15.83:0/3799604360 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd37000d050 con 0x7fd3700130a0 2024-03-25T19:19:46.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.360+0000 7fd3827fc640 1 -- 172.21.15.83:0/3799604360 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fd384004720 con 0x7fd3700130a0 2024-03-25T19:19:46.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.360+0000 7fd3827fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:46.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.360+0000 7fd3827fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:19:46.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.360+0000 7fd3827fc640 10 monclient(hunting): dump: 2024-03-25T19:19:46.360 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T19:19:46.361 INFO:teuthology.orchestra.run.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:46.361 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:46.361 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:46.361 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:46.361 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 3 2024-03-25T19:19:46.361 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:46.361 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:46.361 INFO:teuthology.orchestra.run.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:46.361 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T19:19:46.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.360+0000 7fd3827fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:19:46.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.369+0000 7fd3827fc640 1 -- 172.21.15.83:0/3799604360 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 995900888 0 0) 0x7fd37000cec0 con 0x7fd370007770 2024-03-25T19:19:46.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.369+0000 7fd3827fc640 10 monclient(hunting): _negotiate 2024-03-25T19:19:46.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.369+0000 7fd3827fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:46.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.369+0000 7fd3827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:46.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.369+0000 7fd3827fc640 10 monclient(hunting): my global_id is 4115 2024-03-25T19:19:46.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.369+0000 7fd3827fc640 1 -- 172.21.15.83:0/3799604360 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd37000a820 con 0x7fd370007770 2024-03-25T19:19:46.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.374+0000 7fd3827fc640 1 -- 172.21.15.83:0/3799604360 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 131556707 0 0) 0x7fd37000d050 con 0x7fd3700130a0 2024-03-25T19:19:46.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.374+0000 7fd3827fc640 1 -- 172.21.15.83:0/3799604360 >> v1:172.21.15.83:6790/0 conn(0x7fd370007770 legacy=0x7fd38c141170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:46.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.374+0000 7fd3827fc640 1 -- 172.21.15.83:0/3799604360 >> v1:172.21.15.103:6789/0 conn(0x7fd370008ed0 legacy=0x7fd38c1d7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:46.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.374+0000 7fd3827fc640 10 monclient: _finish_hunting 0 2024-03-25T19:19:46.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.374+0000 7fd3827fc640 1 monclient: found mon.a 2024-03-25T19:19:46.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.374+0000 7fd3827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:19:46.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.374+0000 7fd3827fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:19:46.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.374+0000 7fd3827fc640 1 -- 172.21.15.83:0/3799604360 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd38c141bc0 con 0x7fd3700130a0 2024-03-25T19:19:46.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.374+0000 7fd3827fc640 10 monclient: _finish_auth 0 2024-03-25T19:19:46.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.374+0000 7fd3827fc640 10 monclient: _check_auth_tickets 2024-03-25T19:19:46.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.374+0000 7fd3827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:19:46.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.374+0000 7fd393c63640 5 monclient: authenticate success, global_id 4113 2024-03-25T19:19:46.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.374+0000 7fd393c63640 10 monclient: _renew_subs 2024-03-25T19:19:46.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.374+0000 7fd393c63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:19:46.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.374+0000 7fd393c63640 1 -- 172.21.15.83:0/3799604360 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd38c141a30 con 0x7fd3700130a0 2024-03-25T19:19:46.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.374+0000 7fd393c63640 10 client.4113.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:19:46.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.374+0000 7fd393c63640 10 monclient: _renew_subs 2024-03-25T19:19:46.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.374+0000 7fd393c63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:19:46.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.374+0000 7fd393c63640 1 -- 172.21.15.83:0/3799604360 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd38c141a30 con 0x7fd3700130a0 2024-03-25T19:19:46.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.374+0000 7fd393c63640 1 librados: init done 2024-03-25T19:19:46.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.375+0000 7fd393c63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:46.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.375+0000 7fd393c63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:46.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.375+0000 7fd393c63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:19:46.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.375+0000 7fd393c63640 1 -- 172.21.15.83:0/3799604360 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd38c141a30 con 0x7fd3700130a0 2024-03-25T19:19:46.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.541+0000 7fd3827fc640 1 -- 172.21.15.83:0/3799604360 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd384004b30 con 0x7fd3700130a0 2024-03-25T19:19:46.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.542+0000 7fd3827fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:19:46.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.542+0000 7fd3827fc640 1 -- 172.21.15.83:0/3799604360 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fd384004f40 con 0x7fd3700130a0 2024-03-25T19:19:46.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.542+0000 7fd3827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:46.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.542+0000 7fd3827fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:19:46.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.542+0000 7fd3827fc640 10 monclient: dump: 2024-03-25T19:19:46.542 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T19:19:46.542 INFO:teuthology.orchestra.run.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:46.543 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:46.543 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:46.543 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:46.543 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 3 2024-03-25T19:19:46.543 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:46.543 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:46.543 INFO:teuthology.orchestra.run.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:46.543 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T19:19:46.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.542+0000 7fd3827fc640 1 -- 172.21.15.83:0/3799604360 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fd3840054f0 con 0x7fd3700130a0 2024-03-25T19:19:46.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.542+0000 7fd383fff640 4 set_mon_vals no callback set 2024-03-25T19:19:46.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.542+0000 7fd3827fc640 1 -- 172.21.15.83:0/3799604360 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 3125050352 0 0) 0x7fd384006600 con 0x7fd3700130a0 2024-03-25T19:19:46.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.542+0000 7fd3827fc640 10 client.4113.objecter ms_dispatch 0x7fd38c000b90 osd_map(1..1 src has 1..1) v4 2024-03-25T19:19:46.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.542+0000 7fd3827fc640 3 client.4113.objecter handle_osd_map got epochs [1,1] > 0 2024-03-25T19:19:46.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.542+0000 7fd3827fc640 3 client.4113.objecter handle_osd_map decoding full epoch 1 2024-03-25T19:19:46.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.542+0000 7fd3827fc640 20 client.4113.objecter dump_active .. 0 homeless 2024-03-25T19:19:46.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.542+0000 7fd3827fc640 1 -- 172.21.15.83:0/3799604360 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fd384018dc0 con 0x7fd3700130a0 2024-03-25T19:19:46.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.542+0000 7fd3827fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:46.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.542+0000 7fd3827fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:19:46.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.651+0000 7fd393c63640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:19:46.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.651+0000 7fd393c63640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:19:46.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.651+0000 7fd393c63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:19:46.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.651+0000 7fd393c63640 1 -- 172.21.15.83:0/3799604360 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fd38c141a30 con 0x7fd3700130a0 2024-03-25T19:19:46.793 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-03-25T19:19:46.794+0000 7f02a242b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T19:19:46.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.832+0000 7fd3827fc640 1 -- 172.21.15.83:0/3799604360 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7fd384005ab0 con 0x7fd3700130a0 2024-03-25T19:19:46.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.832+0000 7fd3827fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:19:46.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.832+0000 7fd3827fc640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-25T19:19:46.833 INFO:teuthology.orchestra.run.smithi083.stderr:adjusted tunables profile to default 2024-03-25T19:19:46.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.834+0000 7fd393c63640 10 librados: watch_flush enter 2024-03-25T19:19:46.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.834+0000 7fd393c63640 10 librados: watch_flush exit 2024-03-25T19:19:46.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.834+0000 7fd393c63640 20 client.4113.objecter shutdown clearing up homeless session... 2024-03-25T19:19:46.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.834+0000 7fd393c63640 10 client.4113.objecter successfully canceled tick 2024-03-25T19:19:46.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.834+0000 7fd393c63640 10 monclient: shutdown 2024-03-25T19:19:46.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.834+0000 7fd393c63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:19:46.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.834+0000 7fd393c63640 1 -- 172.21.15.83:0/3799604360 >> v1:172.21.15.83:6789/0 conn(0x7fd3700130a0 legacy=0x7fd38c1e2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:46.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.834+0000 7fd393c63640 1 -- 172.21.15.83:0/3799604360 shutdown_connections 2024-03-25T19:19:46.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.834+0000 7fd393c63640 1 -- 172.21.15.83:0/3799604360 >> 172.21.15.83:0/3799604360 conn(0x7fd38c0b7f40 msgr2=0x7fd38c0b62d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:19:46.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.834+0000 7fd393c63640 1 -- 172.21.15.83:0/3799604360 shutdown_connections 2024-03-25T19:19:46.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.834+0000 7fd393c63640 1 -- 172.21.15.83:0/3799604360 wait complete. 2024-03-25T19:19:46.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:46.834+0000 7fd393c63640 1 librados: shutdown 2024-03-25T19:19:46.838 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-25T19:19:46.838+0000 7f54aa63c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T19:19:46.845 INFO:tasks.ceph:check_enable_crimson: False 2024-03-25T19:19:46.845 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-25T19:19:46.845 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-25T19:19:46.846 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-25T19:19:46.847 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-03-25T19:19:46.848+0000 7f02a242b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T19:19:46.874 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-25T19:19:46.874 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-25T19:19:46.900 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-25T19:19:46.900+0000 7f54aa63c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T19:19:46.926 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-25T19:19:46.926 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-25T19:19:46.994 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-25T19:19:46.994 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-25T19:19:47.063 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-03-25T19:19:47.063 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-25T19:19:47.090 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-03-25T19:19:47.090 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-25T19:19:47.157 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-03-25T19:19:47.157 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-25T19:19:47.224 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-03-25T19:19:47.225 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-25T19:19:47.293 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new 79cfe51b-0994-40c6-b556-3e09b3e0fbf7 0 2024-03-25T19:19:47.433 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.434+0000 7f41e2d9f640 10 monclient: get_monmap_and_config 2024-03-25T19:19:47.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.434+0000 7f41e2d9f640 10 monclient: build_initial_monmap 2024-03-25T19:19:47.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.434+0000 7f41e2d9f640 10 monclient: monmap: 2024-03-25T19:19:47.434 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 0 2024-03-25T19:19:47.434 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:47.434 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:19:47.435189+0000 2024-03-25T19:19:47.434 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:19:47.435189+0000 2024-03-25T19:19:47.434 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 0 (unknown) 2024-03-25T19:19:47.434 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:47.434 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:19:47.434 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:19:47.434 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:19:47.434 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:47.435 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.435+0000 7f41e2d9f640 1 Processor -- start 2024-03-25T19:19:47.436 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.435+0000 7f41e2d9f640 1 -- start start 2024-03-25T19:19:47.436 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.435+0000 7f41e2d9f640 10 monclient: init 2024-03-25T19:19:47.436 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.435+0000 7f41e2d9f640 10 monclient: _reopen_session rank -1 2024-03-25T19:19:47.436 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.435+0000 7f41e2d9f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:19:47.436 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.435+0000 7f41e2d9f640 10 monclient(hunting): picked mon.noname-a con 0x7f41dc06c210 addr v1:172.21.15.83:6789/0 2024-03-25T19:19:47.436 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.435+0000 7f41e2d9f640 10 monclient(hunting): picked mon.noname-c con 0x7f41dc0ae680 addr v1:172.21.15.103:6789/0 2024-03-25T19:19:47.436 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.435+0000 7f41e2d9f640 10 monclient(hunting): picked mon.noname-b con 0x7f41dc0af210 addr v1:172.21.15.83:6790/0 2024-03-25T19:19:47.436 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.435+0000 7f41e2d9f640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41dc0ad9c0 con 0x7f41dc06c210 2024-03-25T19:19:47.436 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.436+0000 7f41e2d9f640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41dc06c5f0 con 0x7f41dc0ae680 2024-03-25T19:19:47.436 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.436+0000 7f41e2d9f640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41dc0adc30 con 0x7f41dc0af210 2024-03-25T19:19:47.436 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.436+0000 7f41e2d9f640 10 monclient(hunting): _renew_subs 2024-03-25T19:19:47.436 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.436+0000 7f41e2d9f640 10 monclient(hunting): authenticate will time out at 6024.570686s 2024-03-25T19:19:47.436 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.436+0000 7f41e1d9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f41dc06c210 0x7f41dc0b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.103:42286/0 (socket says 172.21.15.103:42286) 2024-03-25T19:19:47.436 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.436+0000 7f41e1d9d640 1 -- 172.21.15.103:0/731946732 learned_addr learned my addr 172.21.15.103:0/731946732 (peer_addr_for_me v1:172.21.15.103:0/0) 2024-03-25T19:19:47.437 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-03-25T19:19:47.437+0000 7f02a242b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T19:19:47.437 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.438+0000 7f41e0d9b640 1 -- 172.21.15.103:0/731946732 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 892357969 0 0) 0x7f41dc0ad9c0 con 0x7f41dc06c210 2024-03-25T19:19:47.437 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.438+0000 7f41e0d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:19:47.437 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.438+0000 7f41e0d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:47.437 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.438+0000 7f41e0d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:47.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.438+0000 7f41e0d9b640 10 monclient(hunting): my global_id is 4116 2024-03-25T19:19:47.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.438+0000 7f41e0d9b640 1 -- 172.21.15.103:0/731946732 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41c8003520 con 0x7f41dc06c210 2024-03-25T19:19:47.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.439+0000 7f41e0d9b640 1 -- 172.21.15.103:0/731946732 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2377761115 0 0) 0x7f41dc06c5f0 con 0x7f41dc0ae680 2024-03-25T19:19:47.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.439+0000 7f41e0d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:19:47.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.439+0000 7f41e0d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:47.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.439+0000 7f41e0d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:47.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.439+0000 7f41e0d9b640 10 monclient(hunting): my global_id is 4117 2024-03-25T19:19:47.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.439+0000 7f41e0d9b640 1 -- 172.21.15.103:0/731946732 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41dc0ad9c0 con 0x7f41dc0ae680 2024-03-25T19:19:47.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.439+0000 7f41e0d9b640 1 -- 172.21.15.103:0/731946732 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3986734304 0 0) 0x7f41dc0adc30 con 0x7f41dc0af210 2024-03-25T19:19:47.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.439+0000 7f41e0d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:19:47.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.439+0000 7f41e0d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:47.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.439+0000 7f41e0d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:47.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.439+0000 7f41e0d9b640 10 monclient(hunting): my global_id is 4118 2024-03-25T19:19:47.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.439+0000 7f41e0d9b640 1 -- 172.21.15.103:0/731946732 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41dc06c5f0 con 0x7f41dc0af210 2024-03-25T19:19:47.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.442+0000 7f41e0d9b640 1 -- 172.21.15.103:0/731946732 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 381304207 0 0) 0x7f41c8003520 con 0x7f41dc06c210 2024-03-25T19:19:47.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.442+0000 7f41e0d9b640 1 -- 172.21.15.103:0/731946732 >> v1:172.21.15.83:6790/0 conn(0x7f41dc0af210 legacy=0x7f41dc14dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:47.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.442+0000 7f41e0d9b640 1 -- 172.21.15.103:0/731946732 >> v1:172.21.15.103:6789/0 conn(0x7f41dc0ae680 legacy=0x7f41dc0aeb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:47.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.442+0000 7f41e0d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:19:47.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.442+0000 7f41e0d9b640 1 monclient: found mon.noname-a 2024-03-25T19:19:47.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.442+0000 7f41e0d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:19:47.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.442+0000 7f41e0d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.83:6789/0 2024-03-25T19:19:47.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.442+0000 7f41e0d9b640 1 -- 172.21.15.103:0/731946732 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41dc0adf10 con 0x7f41dc06c210 2024-03-25T19:19:47.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.442+0000 7f41e0d9b640 10 monclient: _finish_auth 0 2024-03-25T19:19:47.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.442+0000 7f41e0d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:19:47.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.442+0000 7f41e0d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:19:47.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.442+0000 7f41e0d9b640 1 -- 172.21.15.103:0/731946732 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f41d8001690 con 0x7f41dc06c210 2024-03-25T19:19:47.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.442+0000 7f41e2d9f640 5 monclient: authenticate success, global_id 4116 2024-03-25T19:19:47.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.442+0000 7f41e2d9f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:19:47.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.442+0000 7f41e0d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:47.443 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.442+0000 7f41e0d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:19:47.443 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.442+0000 7f41e0d9b640 10 monclient: dump: 2024-03-25T19:19:47.443 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 1 2024-03-25T19:19:47.443 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:47.443 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:47.443 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:47.443 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:47.443 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:47.443 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:47.443 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:47.443 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:47.443 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:47.443 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.442+0000 7f41e2d9f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:19:47.445 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.443+0000 7f41e0d9b640 1 -- 172.21.15.103:0/731946732 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41d80017f0 con 0x7f41dc06c210 2024-03-25T19:19:47.445 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.443+0000 7f41e0d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:19:47.445 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.443+0000 7f41e0d9b640 1 -- 172.21.15.103:0/731946732 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f41d8001c00 con 0x7f41dc06c210 2024-03-25T19:19:47.445 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.443+0000 7f41e2d9f640 10 monclient: get_monmap_and_config success 2024-03-25T19:19:47.445 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.443+0000 7f41e2d9f640 4 set_mon_vals no callback set 2024-03-25T19:19:47.445 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.443+0000 7f41e0d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:47.445 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.443+0000 7f41e0d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:19:47.445 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.443+0000 7f41e0d9b640 10 monclient: dump: 2024-03-25T19:19:47.445 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 1 2024-03-25T19:19:47.445 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:47.445 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:47.445 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:47.445 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:47.445 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:47.446 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:47.446 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:47.446 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:47.446 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:47.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.444+0000 7f41e2d9f640 10 monclient: shutdown 2024-03-25T19:19:47.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.444+0000 7f41e2d9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:19:47.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.444+0000 7f41e2d9f640 1 -- 172.21.15.103:0/731946732 >> v1:172.21.15.83:6789/0 conn(0x7f41dc06c210 legacy=0x7f41dc0b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:47.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.444+0000 7f41e2d9f640 1 -- 172.21.15.103:0/731946732 shutdown_connections 2024-03-25T19:19:47.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.444+0000 7f41e2d9f640 1 -- 172.21.15.103:0/731946732 >> 172.21.15.103:0/731946732 conn(0x7f41dc0b5ac0 msgr2=0x7f41dc0b5ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:19:47.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.444+0000 7f41e2d9f640 1 -- 172.21.15.103:0/731946732 shutdown_connections 2024-03-25T19:19:47.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.444+0000 7f41e2d9f640 1 -- 172.21.15.103:0/731946732 wait complete. 2024-03-25T19:19:47.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.445+0000 7f41e2d9f640 10 monclient: build_initial_monmap 2024-03-25T19:19:47.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.445+0000 7f41e2d9f640 10 monclient: monmap: 2024-03-25T19:19:47.446 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 0 2024-03-25T19:19:47.446 INFO:teuthology.orchestra.run.smithi103.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:19:47.446 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 0.000000 2024-03-25T19:19:47.447 INFO:teuthology.orchestra.run.smithi103.stderr:created 0.000000 2024-03-25T19:19:47.447 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 0 (unknown) 2024-03-25T19:19:47.447 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 1 2024-03-25T19:19:47.447 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:19:47.447 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:19:47.447 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:19:47.447 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:47.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.445+0000 7f41e2d9f640 1 librados: starting msgr at 2024-03-25T19:19:47.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.445+0000 7f41e2d9f640 1 librados: starting objecter 2024-03-25T19:19:47.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.445+0000 7f41e2d9f640 1 Processor -- start 2024-03-25T19:19:47.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.445+0000 7f41e2d9f640 1 -- start start 2024-03-25T19:19:47.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.445+0000 7f41e2d9f640 1 librados: setting wanted keys 2024-03-25T19:19:47.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.445+0000 7f41e2d9f640 1 librados: calling monclient init 2024-03-25T19:19:47.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.445+0000 7f41e2d9f640 10 monclient: init 2024-03-25T19:19:47.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.446+0000 7f41e2d9f640 10 monclient: _reopen_session rank -1 2024-03-25T19:19:47.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.446+0000 7f41e2d9f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:19:47.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.446+0000 7f41e2d9f640 10 monclient(hunting): picked mon.noname-b con 0x7f41dc06c210 addr v1:172.21.15.103:6789/0 2024-03-25T19:19:47.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.446+0000 7f41e2d9f640 10 monclient(hunting): picked mon.noname-c con 0x7f41dc0ae680 addr v1:172.21.15.83:6790/0 2024-03-25T19:19:47.450 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.446+0000 7f41e2d9f640 10 monclient(hunting): picked mon.noname-a con 0x7f41dc0af210 addr v1:172.21.15.83:6789/0 2024-03-25T19:19:47.450 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.446+0000 7f41e2d9f640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41dc1c9cf0 con 0x7f41dc0af210 2024-03-25T19:19:47.450 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.446+0000 7f41e2d9f640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41dc1c7d20 con 0x7f41dc06c210 2024-03-25T19:19:47.450 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.446+0000 7f41e2d9f640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41dc1c9f60 con 0x7f41dc0ae680 2024-03-25T19:19:47.450 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.446+0000 7f41e2d9f640 10 monclient(hunting): _renew_subs 2024-03-25T19:19:47.450 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.446+0000 7f41e2d9f640 10 monclient(hunting): authenticate will time out at 6024.580996s 2024-03-25T19:19:47.450 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.446+0000 7f41e159c640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f41dc0ae680 0x7f41dc1c0890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.103:40406/0 (socket says 172.21.15.103:40406) 2024-03-25T19:19:47.450 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.446+0000 7f41e259e640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f41dc0af210 0x7f41dc1c0fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.103:42298/0 (socket says 172.21.15.103:42298) 2024-03-25T19:19:47.450 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.446+0000 7f41e159c640 1 -- 172.21.15.103:0/1565149881 learned_addr learned my addr 172.21.15.103:0/1565149881 (peer_addr_for_me v1:172.21.15.103:0/0) 2024-03-25T19:19:47.450 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.447+0000 7f41d27fc640 10 client.?.objecter ms_handle_connect 0x7f41dc0ae680 2024-03-25T19:19:47.450 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.447+0000 7f41d27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:47.450 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.447+0000 7f41d27fc640 10 client.?.objecter ms_handle_connect 0x7f41dc0af210 2024-03-25T19:19:47.450 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.447+0000 7f41d27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:47.450 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.447+0000 7f41d27fc640 10 client.?.objecter ms_handle_connect 0x7f41dc06c210 2024-03-25T19:19:47.451 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.447+0000 7f41d27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:47.451 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.449+0000 7f41d27fc640 1 -- 172.21.15.103:0/1565149881 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2375540425 0 0) 0x7f41dc1c7d20 con 0x7f41dc06c210 2024-03-25T19:19:47.451 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.449+0000 7f41d27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:19:47.451 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.449+0000 7f41d27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:47.451 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.449+0000 7f41d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:47.451 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.449+0000 7f41d27fc640 10 monclient(hunting): my global_id is 4120 2024-03-25T19:19:47.451 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.449+0000 7f41d27fc640 1 -- 172.21.15.103:0/1565149881 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41c0004eb0 con 0x7f41dc06c210 2024-03-25T19:19:47.451 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.449+0000 7f41d27fc640 1 -- 172.21.15.103:0/1565149881 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3346952252 0 0) 0x7f41dc1c9cf0 con 0x7f41dc0af210 2024-03-25T19:19:47.451 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.449+0000 7f41d27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:19:47.451 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.449+0000 7f41d27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:47.451 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.449+0000 7f41d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:47.451 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.449+0000 7f41d27fc640 10 monclient(hunting): my global_id is 4119 2024-03-25T19:19:47.451 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.449+0000 7f41d27fc640 1 -- 172.21.15.103:0/1565149881 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41dc1c7d20 con 0x7f41dc0af210 2024-03-25T19:19:47.451 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.450+0000 7f41d27fc640 1 -- 172.21.15.103:0/1565149881 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2255272512 0 0) 0x7f41dc1c9f60 con 0x7f41dc0ae680 2024-03-25T19:19:47.451 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.450+0000 7f41d27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:19:47.451 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.450+0000 7f41d27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:47.452 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.450+0000 7f41d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:47.452 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.450+0000 7f41d27fc640 10 monclient(hunting): my global_id is 4121 2024-03-25T19:19:47.452 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.450+0000 7f41d27fc640 1 -- 172.21.15.103:0/1565149881 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41dc1c9cf0 con 0x7f41dc0ae680 2024-03-25T19:19:47.453 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.453+0000 7f41d27fc640 1 -- 172.21.15.103:0/1565149881 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 532488414 0 0) 0x7f41dc1c7d20 con 0x7f41dc0af210 2024-03-25T19:19:47.453 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.453+0000 7f41d27fc640 1 -- 172.21.15.103:0/1565149881 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41dc1c9f60 con 0x7f41dc0af210 2024-03-25T19:19:47.453 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.453+0000 7f41d27fc640 1 -- 172.21.15.103:0/1565149881 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f41d4001bf0 con 0x7f41dc0af210 2024-03-25T19:19:47.453 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.453+0000 7f41d27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:47.453 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.453+0000 7f41d27fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:19:47.453 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.453+0000 7f41d27fc640 10 monclient(hunting): dump: 2024-03-25T19:19:47.453 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 1 2024-03-25T19:19:47.453 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:47.454 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:47.454 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:47.454 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:47.454 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:47.454 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:47.454 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:47.454 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:47.454 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:47.454 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.453+0000 7f41d27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:19:47.454 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.453+0000 7f41d27fc640 1 -- 172.21.15.103:0/1565149881 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3166500586 0 0) 0x7f41c0004eb0 con 0x7f41dc06c210 2024-03-25T19:19:47.454 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.453+0000 7f41d27fc640 1 -- 172.21.15.103:0/1565149881 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41dc1c7d20 con 0x7f41dc06c210 2024-03-25T19:19:47.454 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.453+0000 7f41d27fc640 1 -- 172.21.15.103:0/1565149881 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f41d8001b00 con 0x7f41dc06c210 2024-03-25T19:19:47.454 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.453+0000 7f41d27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:47.454 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.454+0000 7f41d27fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:19:47.454 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.454+0000 7f41d27fc640 10 monclient(hunting): dump: 2024-03-25T19:19:47.454 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 1 2024-03-25T19:19:47.455 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:47.455 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:47.455 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:47.455 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:47.455 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:47.455 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:47.455 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:47.455 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:47.455 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:47.455 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.456+0000 7f41d27fc640 1 -- 172.21.15.103:0/1565149881 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3756408879 0 0) 0x7f41dc1c9cf0 con 0x7f41dc0ae680 2024-03-25T19:19:47.456 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.456+0000 7f41d27fc640 1 -- 172.21.15.103:0/1565149881 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41c0004eb0 con 0x7f41dc0ae680 2024-03-25T19:19:47.456 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.456+0000 7f41d27fc640 1 -- 172.21.15.103:0/1565149881 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f41cc001970 con 0x7f41dc0ae680 2024-03-25T19:19:47.456 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.456+0000 7f41d27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:47.456 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.456+0000 7f41d27fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:19:47.456 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.456+0000 7f41d27fc640 10 monclient(hunting): dump: 2024-03-25T19:19:47.456 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 1 2024-03-25T19:19:47.456 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:47.456 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:47.456 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:47.456 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:47.456 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:47.456 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:47.456 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:47.456 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:47.456 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:47.458 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.457+0000 7f41d27fc640 1 -- 172.21.15.103:0/1565149881 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1104316076 0 0) 0x7f41dc1c9f60 con 0x7f41dc0af210 2024-03-25T19:19:47.458 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.457+0000 7f41d27fc640 1 -- 172.21.15.103:0/1565149881 >> v1:172.21.15.83:6790/0 conn(0x7f41dc0ae680 legacy=0x7f41dc1c0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:47.458 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.457+0000 7f41d27fc640 1 -- 172.21.15.103:0/1565149881 >> v1:172.21.15.103:6789/0 conn(0x7f41dc06c210 legacy=0x7f41dc1c87b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:47.458 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.457+0000 7f41d27fc640 10 monclient: _finish_hunting 0 2024-03-25T19:19:47.458 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.457+0000 7f41d27fc640 1 monclient: found mon.a 2024-03-25T19:19:47.458 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.457+0000 7f41d27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:19:47.458 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.457+0000 7f41d27fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:19:47.458 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.457+0000 7f41d27fc640 1 -- 172.21.15.103:0/1565149881 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41dc1c7890 con 0x7f41dc0af210 2024-03-25T19:19:47.458 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.457+0000 7f41d27fc640 10 monclient: _finish_auth 0 2024-03-25T19:19:47.458 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.457+0000 7f41d27fc640 10 monclient: _check_auth_tickets 2024-03-25T19:19:47.458 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.457+0000 7f41d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:19:47.458 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.457+0000 7f41e2d9f640 5 monclient: authenticate success, global_id 4119 2024-03-25T19:19:47.458 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.457+0000 7f41e2d9f640 10 monclient: _renew_subs 2024-03-25T19:19:47.458 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.457+0000 7f41e2d9f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:19:47.459 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.457+0000 7f41e2d9f640 1 -- 172.21.15.103:0/1565149881 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41dc1ca380 con 0x7f41dc0af210 2024-03-25T19:19:47.459 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.457+0000 7f41e2d9f640 10 client.4119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:19:47.459 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.457+0000 7f41e2d9f640 10 monclient: _renew_subs 2024-03-25T19:19:47.459 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.457+0000 7f41e2d9f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:19:47.459 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.457+0000 7f41e2d9f640 1 -- 172.21.15.103:0/1565149881 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f41dc1c1890 con 0x7f41dc0af210 2024-03-25T19:19:47.459 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.457+0000 7f41e2d9f640 1 librados: init done 2024-03-25T19:19:47.459 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.457+0000 7f41e2d9f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:47.459 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.457+0000 7f41e2d9f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:47.459 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.458+0000 7f41e2d9f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:19:47.459 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.458+0000 7f41e2d9f640 1 -- 172.21.15.103:0/1565149881 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41dc0af950 con 0x7f41dc0af210 2024-03-25T19:19:47.460 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.460+0000 7f41d27fc640 1 -- 172.21.15.103:0/1565149881 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41d40020e0 con 0x7f41dc0af210 2024-03-25T19:19:47.460 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.460+0000 7f41d27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:19:47.460 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.460+0000 7f41d27fc640 1 -- 172.21.15.103:0/1565149881 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f41d4003990 con 0x7f41dc0af210 2024-03-25T19:19:47.460 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.460+0000 7f41d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:47.460 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.460+0000 7f41d27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:19:47.460 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.460+0000 7f41d27fc640 10 monclient: dump: 2024-03-25T19:19:47.460 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 1 2024-03-25T19:19:47.460 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:47.460 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:47.461 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:47.461 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:47.461 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:47.461 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:47.461 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:47.461 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:47.461 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:47.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.460+0000 7f41d27fc640 1 -- 172.21.15.103:0/1565149881 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f41d4003f40 con 0x7f41dc0af210 2024-03-25T19:19:47.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.460+0000 7f41d3fff640 4 set_mon_vals no callback set 2024-03-25T19:19:47.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.460+0000 7f41d27fc640 1 -- 172.21.15.103:0/1565149881 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 3688905830 0 0) 0x7f41d4006690 con 0x7f41dc0af210 2024-03-25T19:19:47.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.460+0000 7f41d27fc640 10 client.4119.objecter ms_dispatch 0x7f41dc000b90 osd_map(2..2 src has 1..2) v4 2024-03-25T19:19:47.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.460+0000 7f41d27fc640 3 client.4119.objecter handle_osd_map got epochs [2,2] > 0 2024-03-25T19:19:47.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.460+0000 7f41d27fc640 3 client.4119.objecter handle_osd_map decoding full epoch 2 2024-03-25T19:19:47.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.461+0000 7f41d27fc640 20 client.4119.objecter dump_active .. 0 homeless 2024-03-25T19:19:47.494 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-03-25T19:19:47.494+0000 7f02a242b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T19:19:47.503 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-25T19:19:47.504+0000 7f54aa63c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T19:19:47.538 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-03-25T19:19:47.539+0000 7f02a242b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T19:19:47.553 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.553+0000 7f41d27fc640 1 -- 172.21.15.103:0/1565149881 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f41d4005b40 con 0x7f41dc0af210 2024-03-25T19:19:47.553 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.553+0000 7f41d27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:47.553 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.553+0000 7f41d27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:19:47.559 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-25T19:19:47.559+0000 7f54aa63c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T19:19:47.594 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-03-25T19:19:47.595+0000 7f02a242b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T19:19:47.603 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-25T19:19:47.604+0000 7f54aa63c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T19:19:47.626 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.627+0000 7f41e2d9f640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "79cfe51b-0994-40c6-b556-3e09b3e0fbf7", "id": 0}] 2024-03-25T19:19:47.626 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.627+0000 7f41e2d9f640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "79cfe51b-0994-40c6-b556-3e09b3e0fbf7", "id": 0}] 2024-03-25T19:19:47.626 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.627+0000 7f41e2d9f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:19:47.626 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.627+0000 7f41e2d9f640 1 -- 172.21.15.103:0/1565149881 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd new", "uuid": "79cfe51b-0994-40c6-b556-3e09b3e0fbf7", "id": 0} v 0) v1 -- 0x7f41dc0627b0 con 0x7f41dc0af210 2024-03-25T19:19:47.638 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-03-25T19:19:47.639+0000 7f02a242b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T19:19:47.658 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-25T19:19:47.658+0000 7f54aa63c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T19:19:47.705 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-25T19:19:47.705+0000 7f54aa63c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T19:19:47.717 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.717+0000 7f41d27fc640 1 -- 172.21.15.103:0/1565149881 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "79cfe51b-0994-40c6-b556-3e09b3e0fbf7", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 1078507747 0 732707515) 0x7f41d4005e10 con 0x7f41dc0af210 2024-03-25T19:19:47.717 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.717+0000 7f41d27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "79cfe51b-0994-40c6-b556-3e09b3e0fbf7", "id": 0}] 2024-03-25T19:19:47.717 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.717+0000 7f41d27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:19:47.717 INFO:teuthology.orchestra.run.smithi103.stdout:0 2024-03-25T19:19:47.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.718+0000 7f41e2d9f640 10 librados: watch_flush enter 2024-03-25T19:19:47.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.718+0000 7f41e2d9f640 10 librados: watch_flush exit 2024-03-25T19:19:47.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.718+0000 7f41e2d9f640 20 client.4119.objecter shutdown clearing up homeless session... 2024-03-25T19:19:47.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.718+0000 7f41e2d9f640 10 client.4119.objecter successfully canceled tick 2024-03-25T19:19:47.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.718+0000 7f41e2d9f640 10 monclient: shutdown 2024-03-25T19:19:47.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.718+0000 7f41e2d9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:19:47.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.718+0000 7f41e2d9f640 1 -- 172.21.15.103:0/1565149881 >> v1:172.21.15.83:6789/0 conn(0x7f41dc0af210 legacy=0x7f41dc1c0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:47.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.718+0000 7f41e2d9f640 1 -- 172.21.15.103:0/1565149881 shutdown_connections 2024-03-25T19:19:47.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.718+0000 7f41e2d9f640 1 -- 172.21.15.103:0/1565149881 >> 172.21.15.103:0/1565149881 conn(0x7f41dc0b5ac0 msgr2=0x7f41dc0b50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:19:47.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.718+0000 7f41e2d9f640 1 -- 172.21.15.103:0/1565149881 shutdown_connections 2024-03-25T19:19:47.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.718+0000 7f41e2d9f640 1 -- 172.21.15.103:0/1565149881 wait complete. 2024-03-25T19:19:47.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.718+0000 7f41e2d9f640 1 librados: shutdown 2024-03-25T19:19:47.728 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new b51f462f-9495-4b42-94a1-cfa31e1db349 1 2024-03-25T19:19:47.759 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-03-25T19:19:47.760+0000 7f02a242b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T19:19:47.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.827+0000 7f19f67ba640 10 monclient: get_monmap_and_config 2024-03-25T19:19:47.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.827+0000 7f19f67ba640 10 monclient: build_initial_monmap 2024-03-25T19:19:47.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.827+0000 7f19f67ba640 10 monclient: monmap: 2024-03-25T19:19:47.827 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 0 2024-03-25T19:19:47.827 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:47.827 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:19:47.828323+0000 2024-03-25T19:19:47.827 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:19:47.828323+0000 2024-03-25T19:19:47.827 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 0 (unknown) 2024-03-25T19:19:47.827 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:47.827 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:19:47.827 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:19:47.827 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:19:47.828 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:47.829 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.828+0000 7f19f67ba640 1 Processor -- start 2024-03-25T19:19:47.829 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.828+0000 7f19f67ba640 1 -- start start 2024-03-25T19:19:47.829 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.828+0000 7f19f67ba640 10 monclient: init 2024-03-25T19:19:47.829 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.829+0000 7f19f67ba640 10 monclient: _reopen_session rank -1 2024-03-25T19:19:47.829 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.829+0000 7f19f67ba640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:19:47.829 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.829+0000 7f19f67ba640 10 monclient(hunting): picked mon.noname-b con 0x7f19f00b4700 addr v1:172.21.15.83:6790/0 2024-03-25T19:19:47.829 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.829+0000 7f19f67ba640 10 monclient(hunting): picked mon.noname-c con 0x7f19f0142a20 addr v1:172.21.15.103:6789/0 2024-03-25T19:19:47.830 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.829+0000 7f19f67ba640 10 monclient(hunting): picked mon.noname-a con 0x7f19f00a96b0 addr v1:172.21.15.83:6789/0 2024-03-25T19:19:47.830 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.829+0000 7f19f67ba640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19f00a9d60 con 0x7f19f00a96b0 2024-03-25T19:19:47.830 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.829+0000 7f19f67ba640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19f013ff10 con 0x7f19f0142a20 2024-03-25T19:19:47.830 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.829+0000 7f19f67ba640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19f00aa1f0 con 0x7f19f00b4700 2024-03-25T19:19:47.830 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.829+0000 7f19f67ba640 10 monclient(hunting): _renew_subs 2024-03-25T19:19:47.830 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.829+0000 7f19f67ba640 10 monclient(hunting): authenticate will time out at 6024.963864s 2024-03-25T19:19:47.830 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.829+0000 7f19ef7fe640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f19f0142a20 0x7f19f013f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:33374/0 (socket says 172.21.15.103:33374) 2024-03-25T19:19:47.830 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.829+0000 7f19ef7fe640 1 -- 172.21.15.103:0/3567359302 learned_addr learned my addr 172.21.15.103:0/3567359302 (peer_addr_for_me v1:172.21.15.103:0/0) 2024-03-25T19:19:47.831 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.832+0000 7f19eeffd640 1 -- 172.21.15.103:0/3567359302 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3172161422 0 0) 0x7f19f00a9d60 con 0x7f19f00a96b0 2024-03-25T19:19:47.831 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.832+0000 7f19eeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:19:47.831 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.832+0000 7f19eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:47.831 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.832+0000 7f19eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:47.831 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.832+0000 7f19eeffd640 10 monclient(hunting): my global_id is 4122 2024-03-25T19:19:47.831 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.832+0000 7f19eeffd640 1 -- 172.21.15.103:0/3567359302 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19d8003520 con 0x7f19f00a96b0 2024-03-25T19:19:47.832 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.832+0000 7f19eeffd640 1 -- 172.21.15.103:0/3567359302 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 185146463 0 0) 0x7f19f013ff10 con 0x7f19f0142a20 2024-03-25T19:19:47.832 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.832+0000 7f19eeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:19:47.832 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.832+0000 7f19eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:47.832 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.832+0000 7f19eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:47.832 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.832+0000 7f19eeffd640 10 monclient(hunting): my global_id is 4123 2024-03-25T19:19:47.833 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.832+0000 7f19eeffd640 1 -- 172.21.15.103:0/3567359302 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19f00a9d60 con 0x7f19f0142a20 2024-03-25T19:19:47.833 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.832+0000 7f19eeffd640 1 -- 172.21.15.103:0/3567359302 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 443492579 0 0) 0x7f19f00aa1f0 con 0x7f19f00b4700 2024-03-25T19:19:47.833 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.832+0000 7f19eeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:19:47.833 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.832+0000 7f19eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:47.833 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.832+0000 7f19eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:47.833 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.832+0000 7f19eeffd640 10 monclient(hunting): my global_id is 4124 2024-03-25T19:19:47.833 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.832+0000 7f19eeffd640 1 -- 172.21.15.103:0/3567359302 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19f013ff10 con 0x7f19f00b4700 2024-03-25T19:19:47.833 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-03-25T19:19:47.834+0000 7f02a242b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T19:19:47.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.835+0000 7f19eeffd640 1 -- 172.21.15.103:0/3567359302 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2547130602 0 0) 0x7f19d8003520 con 0x7f19f00a96b0 2024-03-25T19:19:47.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.835+0000 7f19eeffd640 1 -- 172.21.15.103:0/3567359302 >> v1:172.21.15.83:6790/0 conn(0x7f19f00b4700 legacy=0x7f19f0142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:47.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.835+0000 7f19eeffd640 1 -- 172.21.15.103:0/3567359302 >> v1:172.21.15.103:6789/0 conn(0x7f19f0142a20 legacy=0x7f19f013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:47.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.835+0000 7f19eeffd640 10 monclient: _finish_hunting 0 2024-03-25T19:19:47.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.835+0000 7f19eeffd640 1 monclient: found mon.noname-a 2024-03-25T19:19:47.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.835+0000 7f19eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:19:47.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.835+0000 7f19eeffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.83:6789/0 2024-03-25T19:19:47.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.835+0000 7f19eeffd640 1 -- 172.21.15.103:0/3567359302 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f00b4ae0 con 0x7f19f00a96b0 2024-03-25T19:19:47.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.835+0000 7f19eeffd640 10 monclient: _finish_auth 0 2024-03-25T19:19:47.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.835+0000 7f19eeffd640 10 monclient: _check_auth_tickets 2024-03-25T19:19:47.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.835+0000 7f19eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:19:47.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.835+0000 7f19eeffd640 1 -- 172.21.15.103:0/3567359302 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f19e8001a10 con 0x7f19f00a96b0 2024-03-25T19:19:47.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.835+0000 7f19f67ba640 5 monclient: authenticate success, global_id 4122 2024-03-25T19:19:47.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.835+0000 7f19f67ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:19:47.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.835+0000 7f19eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:47.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.835+0000 7f19eeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:19:47.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.835+0000 7f19eeffd640 10 monclient: dump: 2024-03-25T19:19:47.836 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 1 2024-03-25T19:19:47.836 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:47.836 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:47.836 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:47.836 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:47.836 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:47.836 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:47.836 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:47.836 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:47.836 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:47.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.835+0000 7f19f67ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:19:47.838 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.837+0000 7f19eeffd640 1 -- 172.21.15.103:0/3567359302 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19e8001b70 con 0x7f19f00a96b0 2024-03-25T19:19:47.838 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.837+0000 7f19eeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:19:47.838 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.837+0000 7f19eeffd640 1 -- 172.21.15.103:0/3567359302 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f19e8001f80 con 0x7f19f00a96b0 2024-03-25T19:19:47.838 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.837+0000 7f19f67ba640 10 monclient: get_monmap_and_config success 2024-03-25T19:19:47.838 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.837+0000 7f19f67ba640 4 set_mon_vals no callback set 2024-03-25T19:19:47.838 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.837+0000 7f19eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:47.838 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.837+0000 7f19eeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:19:47.838 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.837+0000 7f19eeffd640 10 monclient: dump: 2024-03-25T19:19:47.838 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 1 2024-03-25T19:19:47.838 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:47.839 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:47.839 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:47.839 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:47.839 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:47.839 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:47.839 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:47.839 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:47.839 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:47.839 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.837+0000 7f19f67ba640 10 monclient: shutdown 2024-03-25T19:19:47.839 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.837+0000 7f19f67ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:19:47.839 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.837+0000 7f19f67ba640 1 -- 172.21.15.103:0/3567359302 >> v1:172.21.15.83:6789/0 conn(0x7f19f00a96b0 legacy=0x7f19f013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:47.839 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.837+0000 7f19f67ba640 1 -- 172.21.15.103:0/3567359302 shutdown_connections 2024-03-25T19:19:47.839 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.837+0000 7f19f67ba640 1 -- 172.21.15.103:0/3567359302 >> 172.21.15.103:0/3567359302 conn(0x7f19f00668e0 msgr2=0x7f19f0066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:19:47.839 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.837+0000 7f19f67ba640 1 -- 172.21.15.103:0/3567359302 shutdown_connections 2024-03-25T19:19:47.839 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.837+0000 7f19f67ba640 1 -- 172.21.15.103:0/3567359302 wait complete. 2024-03-25T19:19:47.840 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.838+0000 7f19f67ba640 10 monclient: build_initial_monmap 2024-03-25T19:19:47.840 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.838+0000 7f19f67ba640 10 monclient: monmap: 2024-03-25T19:19:47.840 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 0 2024-03-25T19:19:47.840 INFO:teuthology.orchestra.run.smithi103.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:19:47.840 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 0.000000 2024-03-25T19:19:47.840 INFO:teuthology.orchestra.run.smithi103.stderr:created 0.000000 2024-03-25T19:19:47.840 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 0 (unknown) 2024-03-25T19:19:47.840 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 1 2024-03-25T19:19:47.840 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:19:47.840 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:19:47.840 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:19:47.840 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:47.840 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.838+0000 7f19f67ba640 1 librados: starting msgr at 2024-03-25T19:19:47.840 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.838+0000 7f19f67ba640 1 librados: starting objecter 2024-03-25T19:19:47.840 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.838+0000 7f19f67ba640 1 Processor -- start 2024-03-25T19:19:47.840 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.839+0000 7f19f67ba640 1 -- start start 2024-03-25T19:19:47.840 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.839+0000 7f19f67ba640 1 librados: setting wanted keys 2024-03-25T19:19:47.840 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.839+0000 7f19f67ba640 1 librados: calling monclient init 2024-03-25T19:19:47.841 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.839+0000 7f19f67ba640 10 monclient: init 2024-03-25T19:19:47.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.839+0000 7f19f67ba640 10 monclient: _reopen_session rank -1 2024-03-25T19:19:47.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.839+0000 7f19f67ba640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:19:47.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.839+0000 7f19f67ba640 10 monclient(hunting): picked mon.noname-c con 0x7f19f00a96b0 addr v1:172.21.15.83:6790/0 2024-03-25T19:19:47.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.839+0000 7f19f67ba640 10 monclient(hunting): picked mon.noname-a con 0x7f19f00b4700 addr v1:172.21.15.83:6789/0 2024-03-25T19:19:47.843 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.839+0000 7f19f67ba640 10 monclient(hunting): picked mon.noname-b con 0x7f19f0142a20 addr v1:172.21.15.103:6789/0 2024-03-25T19:19:47.843 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.839+0000 7f19f67ba640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19f00ac830 con 0x7f19f00b4700 2024-03-25T19:19:47.843 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.839+0000 7f19f67ba640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19f013cc10 con 0x7f19f0142a20 2024-03-25T19:19:47.843 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.839+0000 7f19f67ba640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19f013b7f0 con 0x7f19f00a96b0 2024-03-25T19:19:47.843 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.839+0000 7f19f67ba640 10 monclient(hunting): _renew_subs 2024-03-25T19:19:47.844 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.839+0000 7f19f67ba640 10 monclient(hunting): authenticate will time out at 6024.974256s 2024-03-25T19:19:47.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.840+0000 7f19effff640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f19f00a96b0 0x7f19f013c500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.103:40420/0 (socket says 172.21.15.103:40420) 2024-03-25T19:19:47.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.840+0000 7f19effff640 1 -- 172.21.15.103:0/562529475 learned_addr learned my addr 172.21.15.103:0/562529475 (peer_addr_for_me v1:172.21.15.103:0/0) 2024-03-25T19:19:47.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.840+0000 7f19ecff9640 10 client.?.objecter ms_handle_connect 0x7f19f00a96b0 2024-03-25T19:19:47.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.840+0000 7f19ecff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:47.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.840+0000 7f19ecff9640 10 client.?.objecter ms_handle_connect 0x7f19f0142a20 2024-03-25T19:19:47.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.840+0000 7f19ecff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:47.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.840+0000 7f19ecff9640 10 client.?.objecter ms_handle_connect 0x7f19f00b4700 2024-03-25T19:19:47.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.840+0000 7f19ecff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:47.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.842+0000 7f19ecff9640 1 -- 172.21.15.103:0/562529475 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2417825575 0 0) 0x7f19f00ac830 con 0x7f19f00b4700 2024-03-25T19:19:47.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.842+0000 7f19ecff9640 10 monclient(hunting): _negotiate 2024-03-25T19:19:47.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.842+0000 7f19ecff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:47.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.842+0000 7f19ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:47.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.842+0000 7f19ecff9640 10 monclient(hunting): my global_id is 4125 2024-03-25T19:19:47.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.842+0000 7f19ecff9640 1 -- 172.21.15.103:0/562529475 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19dc004eb0 con 0x7f19f00b4700 2024-03-25T19:19:47.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.842+0000 7f19ecff9640 1 -- 172.21.15.103:0/562529475 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1476002097 0 0) 0x7f19f013cc10 con 0x7f19f0142a20 2024-03-25T19:19:47.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.842+0000 7f19ecff9640 10 monclient(hunting): _negotiate 2024-03-25T19:19:47.846 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.842+0000 7f19ecff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:47.846 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.842+0000 7f19ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:47.846 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.842+0000 7f19ecff9640 10 monclient(hunting): my global_id is 4126 2024-03-25T19:19:47.846 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.842+0000 7f19ecff9640 1 -- 172.21.15.103:0/562529475 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19f00ac830 con 0x7f19f0142a20 2024-03-25T19:19:47.846 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.843+0000 7f19ecff9640 1 -- 172.21.15.103:0/562529475 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1138818482 0 0) 0x7f19f013b7f0 con 0x7f19f00a96b0 2024-03-25T19:19:47.846 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.843+0000 7f19ecff9640 10 monclient(hunting): _negotiate 2024-03-25T19:19:47.846 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.843+0000 7f19ecff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:47.846 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.843+0000 7f19ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:47.846 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.843+0000 7f19ecff9640 10 monclient(hunting): my global_id is 4127 2024-03-25T19:19:47.846 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.843+0000 7f19ecff9640 1 -- 172.21.15.103:0/562529475 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19f013cc10 con 0x7f19f00a96b0 2024-03-25T19:19:47.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.847+0000 7f19ecff9640 1 -- 172.21.15.103:0/562529475 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3761884081 0 0) 0x7f19dc004eb0 con 0x7f19f00b4700 2024-03-25T19:19:47.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.847+0000 7f19ecff9640 1 -- 172.21.15.103:0/562529475 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19f013b7f0 con 0x7f19f00b4700 2024-03-25T19:19:47.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.847+0000 7f19ecff9640 1 -- 172.21.15.103:0/562529475 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f19e0001910 con 0x7f19f00b4700 2024-03-25T19:19:47.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.847+0000 7f19ecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:47.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.847+0000 7f19ecff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:19:47.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.847+0000 7f19ecff9640 10 monclient(hunting): dump: 2024-03-25T19:19:47.847 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 1 2024-03-25T19:19:47.847 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:47.847 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:47.847 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:47.848 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:47.848 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:47.848 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:47.848 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:47.848 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:47.848 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:47.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.847+0000 7f19ecff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:19:47.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.847+0000 7f19ecff9640 1 -- 172.21.15.103:0/562529475 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3249125857 0 0) 0x7f19f00ac830 con 0x7f19f0142a20 2024-03-25T19:19:47.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.847+0000 7f19ecff9640 1 -- 172.21.15.103:0/562529475 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19dc004eb0 con 0x7f19f0142a20 2024-03-25T19:19:47.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.847+0000 7f19ecff9640 1 -- 172.21.15.103:0/562529475 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f19e8001e80 con 0x7f19f0142a20 2024-03-25T19:19:47.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.847+0000 7f19ecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:47.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.847+0000 7f19ecff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:19:47.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.847+0000 7f19ecff9640 10 monclient(hunting): dump: 2024-03-25T19:19:47.848 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 1 2024-03-25T19:19:47.848 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:47.848 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:47.848 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:47.849 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:47.849 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:47.849 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:47.849 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:47.849 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:47.849 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:47.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.849+0000 7f19ecff9640 1 -- 172.21.15.103:0/562529475 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4112690831 0 0) 0x7f19f013cc10 con 0x7f19f00a96b0 2024-03-25T19:19:47.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.849+0000 7f19ecff9640 1 -- 172.21.15.103:0/562529475 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19f00ac830 con 0x7f19f00a96b0 2024-03-25T19:19:47.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.849+0000 7f19ecff9640 1 -- 172.21.15.103:0/562529475 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f19e40019b0 con 0x7f19f00a96b0 2024-03-25T19:19:47.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.849+0000 7f19ecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:47.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.849+0000 7f19ecff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:19:47.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.849+0000 7f19ecff9640 10 monclient(hunting): dump: 2024-03-25T19:19:47.851 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 1 2024-03-25T19:19:47.851 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:47.851 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:47.851 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:47.851 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:47.851 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:47.851 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:47.851 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:47.851 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:47.851 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:47.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.850+0000 7f19ecff9640 1 -- 172.21.15.103:0/562529475 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 550237759 0 0) 0x7f19f013b7f0 con 0x7f19f00b4700 2024-03-25T19:19:47.852 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.850+0000 7f19ecff9640 1 -- 172.21.15.103:0/562529475 >> v1:172.21.15.83:6790/0 conn(0x7f19f00a96b0 legacy=0x7f19f013c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:47.852 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.850+0000 7f19ecff9640 1 -- 172.21.15.103:0/562529475 >> v1:172.21.15.103:6789/0 conn(0x7f19f0142a20 legacy=0x7f19f01e23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:47.852 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.850+0000 7f19ecff9640 10 monclient: _finish_hunting 0 2024-03-25T19:19:47.852 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.850+0000 7f19ecff9640 1 monclient: found mon.a 2024-03-25T19:19:47.852 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.850+0000 7f19ecff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:19:47.852 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.850+0000 7f19ecff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:19:47.852 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.850+0000 7f19ecff9640 1 -- 172.21.15.103:0/562529475 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f013b980 con 0x7f19f00b4700 2024-03-25T19:19:47.852 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.850+0000 7f19ecff9640 10 monclient: _finish_auth 0 2024-03-25T19:19:47.852 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.850+0000 7f19ecff9640 10 monclient: _check_auth_tickets 2024-03-25T19:19:47.852 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.850+0000 7f19ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:19:47.852 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.850+0000 7f19f67ba640 5 monclient: authenticate success, global_id 4125 2024-03-25T19:19:47.852 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.850+0000 7f19f67ba640 10 monclient: _renew_subs 2024-03-25T19:19:47.852 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.850+0000 7f19f67ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:19:47.852 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.850+0000 7f19f67ba640 1 -- 172.21.15.103:0/562529475 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19f00ad360 con 0x7f19f00b4700 2024-03-25T19:19:47.852 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.850+0000 7f19f67ba640 10 client.4125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:19:47.852 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.850+0000 7f19f67ba640 10 monclient: _renew_subs 2024-03-25T19:19:47.852 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.850+0000 7f19f67ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:19:47.853 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.850+0000 7f19f67ba640 1 -- 172.21.15.103:0/562529475 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f19f013a520 con 0x7f19f00b4700 2024-03-25T19:19:47.853 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.850+0000 7f19f67ba640 1 librados: init done 2024-03-25T19:19:47.853 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.851+0000 7f19f67ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:47.853 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.851+0000 7f19f67ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:47.853 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.851+0000 7f19f67ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:19:47.853 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.851+0000 7f19f67ba640 1 -- 172.21.15.103:0/562529475 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19f00ac630 con 0x7f19f00b4700 2024-03-25T19:19:47.853 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.854+0000 7f19ecff9640 1 -- 172.21.15.103:0/562529475 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19e00047e0 con 0x7f19f00b4700 2024-03-25T19:19:47.853 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.854+0000 7f19ecff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:19:47.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.854+0000 7f19ecff9640 1 -- 172.21.15.103:0/562529475 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f19e0004af0 con 0x7f19f00b4700 2024-03-25T19:19:47.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.854+0000 7f19ecff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:47.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.854+0000 7f19ecff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:19:47.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.854+0000 7f19ecff9640 10 monclient: dump: 2024-03-25T19:19:47.854 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 1 2024-03-25T19:19:47.854 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:47.854 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:47.854 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:47.854 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:47.854 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:47.854 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:47.854 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:47.854 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:47.854 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:47.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.854+0000 7f19ecff9640 1 -- 172.21.15.103:0/562529475 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f19e0004fe0 con 0x7f19f00b4700 2024-03-25T19:19:47.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.854+0000 7f19ecff9640 1 -- 172.21.15.103:0/562529475 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 1577432841 0 0) 0x7f19e0006540 con 0x7f19f00b4700 2024-03-25T19:19:47.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.854+0000 7f19ecff9640 10 client.4125.objecter ms_dispatch 0x7f19f0000b90 osd_map(3..3 src has 1..3) v4 2024-03-25T19:19:47.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.854+0000 7f19ecff9640 3 client.4125.objecter handle_osd_map got epochs [3,3] > 0 2024-03-25T19:19:47.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.854+0000 7f19ecff9640 3 client.4125.objecter handle_osd_map decoding full epoch 3 2024-03-25T19:19:47.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.854+0000 7f19ee7fc640 4 set_mon_vals no callback set 2024-03-25T19:19:47.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.854+0000 7f19ecff9640 20 client.4125.objecter dump_active .. 0 homeless 2024-03-25T19:19:47.859 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-25T19:19:47.860+0000 7f54aa63c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T19:19:47.884 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-03-25T19:19:47.885+0000 7f02a242b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T19:19:47.933 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-25T19:19:47.933+0000 7f54aa63c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T19:19:47.936 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-03-25T19:19:47.936+0000 7f02a242b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T19:19:47.948 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.949+0000 7f19ecff9640 1 -- 172.21.15.103:0/562529475 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f19e0006920 con 0x7f19f00b4700 2024-03-25T19:19:47.948 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.949+0000 7f19ecff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:47.948 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:47.949+0000 7f19ecff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:19:47.980 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-25T19:19:47.980+0000 7f54aa63c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T19:19:47.984 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-03-25T19:19:47.985+0000 7f02a242b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T19:19:48.020 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.020+0000 7f19f67ba640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b51f462f-9495-4b42-94a1-cfa31e1db349", "id": 1}] 2024-03-25T19:19:48.020 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.020+0000 7f19f67ba640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b51f462f-9495-4b42-94a1-cfa31e1db349", "id": 1}] 2024-03-25T19:19:48.020 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.020+0000 7f19f67ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:19:48.020 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.020+0000 7f19f67ba640 1 -- 172.21.15.103:0/562529475 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd new", "uuid": "b51f462f-9495-4b42-94a1-cfa31e1db349", "id": 1} v 0) v1 -- 0x7f19f004f730 con 0x7f19f00b4700 2024-03-25T19:19:48.033 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-25T19:19:48.033+0000 7f54aa63c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T19:19:48.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.073+0000 7f19ecff9640 1 -- 172.21.15.103:0/562529475 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b51f462f-9495-4b42-94a1-cfa31e1db349", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 3426334304 0 940483276) 0x7f19e0006b90 con 0x7f19f00b4700 2024-03-25T19:19:48.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.073+0000 7f19ecff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b51f462f-9495-4b42-94a1-cfa31e1db349", "id": 1}] 2024-03-25T19:19:48.073 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.073+0000 7f19ecff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:19:48.073 INFO:teuthology.orchestra.run.smithi103.stdout:1 2024-03-25T19:19:48.074 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.073+0000 7f19f67ba640 10 librados: watch_flush enter 2024-03-25T19:19:48.075 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.073+0000 7f19f67ba640 10 librados: watch_flush exit 2024-03-25T19:19:48.075 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.073+0000 7f19f67ba640 20 client.4125.objecter shutdown clearing up homeless session... 2024-03-25T19:19:48.075 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.073+0000 7f19f67ba640 10 client.4125.objecter successfully canceled tick 2024-03-25T19:19:48.075 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.074+0000 7f19f67ba640 10 monclient: shutdown 2024-03-25T19:19:48.075 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.074+0000 7f19f67ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:19:48.075 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.074+0000 7f19f67ba640 1 -- 172.21.15.103:0/562529475 >> v1:172.21.15.83:6789/0 conn(0x7f19f00b4700 legacy=0x7f19f00acc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:48.075 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.074+0000 7f19f67ba640 1 -- 172.21.15.103:0/562529475 shutdown_connections 2024-03-25T19:19:48.075 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.074+0000 7f19f67ba640 1 -- 172.21.15.103:0/562529475 >> 172.21.15.103:0/562529475 conn(0x7f19f00668e0 msgr2=0x7f19f00b6f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:19:48.075 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.074+0000 7f19f67ba640 1 -- 172.21.15.103:0/562529475 shutdown_connections 2024-03-25T19:19:48.075 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.074+0000 7f19f67ba640 1 -- 172.21.15.103:0/562529475 wait complete. 2024-03-25T19:19:48.075 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.074+0000 7f19f67ba640 1 librados: shutdown 2024-03-25T19:19:48.084 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new 467ce07c-8165-4299-90b2-274bc771cff5 2 2024-03-25T19:19:48.091 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-25T19:19:48.092+0000 7f54aa63c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T19:19:48.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.225+0000 7f67adcd3640 10 monclient: get_monmap_and_config 2024-03-25T19:19:48.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.225+0000 7f67adcd3640 10 monclient: build_initial_monmap 2024-03-25T19:19:48.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.225+0000 7f67adcd3640 10 monclient: monmap: 2024-03-25T19:19:48.225 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 0 2024-03-25T19:19:48.225 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:48.225 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:19:48.226262+0000 2024-03-25T19:19:48.225 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:19:48.226262+0000 2024-03-25T19:19:48.225 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 0 (unknown) 2024-03-25T19:19:48.225 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:48.225 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:19:48.225 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:19:48.225 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:19:48.226 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:48.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.226+0000 7f67adcd3640 1 Processor -- start 2024-03-25T19:19:48.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.226+0000 7f67adcd3640 1 -- start start 2024-03-25T19:19:48.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.226+0000 7f67adcd3640 10 monclient: init 2024-03-25T19:19:48.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.227+0000 7f67adcd3640 10 monclient: _reopen_session rank -1 2024-03-25T19:19:48.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.227+0000 7f67adcd3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:19:48.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.227+0000 7f67adcd3640 10 monclient(hunting): picked mon.noname-a con 0x7f67a80b4700 addr v1:172.21.15.83:6789/0 2024-03-25T19:19:48.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.227+0000 7f67adcd3640 10 monclient(hunting): picked mon.noname-c con 0x7f67a8142a20 addr v1:172.21.15.103:6789/0 2024-03-25T19:19:48.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.227+0000 7f67adcd3640 10 monclient(hunting): picked mon.noname-b con 0x7f67a80a96b0 addr v1:172.21.15.83:6790/0 2024-03-25T19:19:48.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.227+0000 7f67adcd3640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67a80a9d60 con 0x7f67a80b4700 2024-03-25T19:19:48.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.227+0000 7f67adcd3640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67a813ff10 con 0x7f67a8142a20 2024-03-25T19:19:48.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.227+0000 7f67adcd3640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67a80aa1f0 con 0x7f67a80a96b0 2024-03-25T19:19:48.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.227+0000 7f67adcd3640 10 monclient(hunting): _renew_subs 2024-03-25T19:19:48.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.227+0000 7f67adcd3640 10 monclient(hunting): authenticate will time out at 6025.361834s 2024-03-25T19:19:48.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.227+0000 7f67a7fff640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f67a80a96b0 0x7f67a813f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.103:40436/0 (socket says 172.21.15.103:40436) 2024-03-25T19:19:48.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.227+0000 7f67a77fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f67a80b4700 0x7f67a8142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.103:42324/0 (socket says 172.21.15.103:42324) 2024-03-25T19:19:48.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.227+0000 7f67a7fff640 1 -- 172.21.15.103:0/4218864120 learned_addr learned my addr 172.21.15.103:0/4218864120 (peer_addr_for_me v1:172.21.15.103:0/0) 2024-03-25T19:19:48.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.230+0000 7f67a6ffd640 1 -- 172.21.15.103:0/4218864120 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2314142048 0 0) 0x7f67a80aa1f0 con 0x7f67a80a96b0 2024-03-25T19:19:48.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.230+0000 7f67a6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:19:48.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.230+0000 7f67a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:48.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.230+0000 7f67a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:48.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.230+0000 7f67a6ffd640 10 monclient(hunting): my global_id is 4130 2024-03-25T19:19:48.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.230+0000 7f67a6ffd640 1 -- 172.21.15.103:0/4218864120 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6794003520 con 0x7f67a80a96b0 2024-03-25T19:19:48.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.230+0000 7f67a6ffd640 1 -- 172.21.15.103:0/4218864120 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1005571169 0 0) 0x7f67a813ff10 con 0x7f67a8142a20 2024-03-25T19:19:48.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.230+0000 7f67a6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:19:48.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.230+0000 7f67a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:48.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.230+0000 7f67a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:48.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.230+0000 7f67a6ffd640 10 monclient(hunting): my global_id is 4129 2024-03-25T19:19:48.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.230+0000 7f67a6ffd640 1 -- 172.21.15.103:0/4218864120 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67a80aa1f0 con 0x7f67a8142a20 2024-03-25T19:19:48.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.230+0000 7f67a6ffd640 1 -- 172.21.15.103:0/4218864120 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2115193844 0 0) 0x7f67a80a9d60 con 0x7f67a80b4700 2024-03-25T19:19:48.231 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.230+0000 7f67a6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:19:48.231 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.230+0000 7f67a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:48.231 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.230+0000 7f67a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:48.231 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.230+0000 7f67a6ffd640 10 monclient(hunting): my global_id is 4128 2024-03-25T19:19:48.231 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.230+0000 7f67a6ffd640 1 -- 172.21.15.103:0/4218864120 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67a813ff10 con 0x7f67a80b4700 2024-03-25T19:19:48.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.233+0000 7f67a6ffd640 1 -- 172.21.15.103:0/4218864120 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2847426280 0 0) 0x7f67a80aa1f0 con 0x7f67a8142a20 2024-03-25T19:19:48.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.233+0000 7f67a6ffd640 1 -- 172.21.15.103:0/4218864120 >> v1:172.21.15.83:6790/0 conn(0x7f67a80a96b0 legacy=0x7f67a813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:48.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.233+0000 7f67a6ffd640 1 -- 172.21.15.103:0/4218864120 >> v1:172.21.15.83:6789/0 conn(0x7f67a80b4700 legacy=0x7f67a8142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:48.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.233+0000 7f67a6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:19:48.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.233+0000 7f67a6ffd640 1 monclient: found mon.noname-c 2024-03-25T19:19:48.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.233+0000 7f67a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:19:48.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.233+0000 7f67a6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.103:6789/0 2024-03-25T19:19:48.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.233+0000 7f67a6ffd640 1 -- 172.21.15.103:0/4218864120 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67a80b4ae0 con 0x7f67a8142a20 2024-03-25T19:19:48.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.233+0000 7f67a6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:19:48.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.233+0000 7f67a6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:19:48.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.233+0000 7f67a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:19:48.234 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.233+0000 7f67a6ffd640 1 -- 172.21.15.103:0/4218864120 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f6790001670 con 0x7f67a8142a20 2024-03-25T19:19:48.234 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.233+0000 7f67a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:48.234 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.233+0000 7f67a6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:19:48.234 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.233+0000 7f67a6ffd640 10 monclient: dump: 2024-03-25T19:19:48.234 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 1 2024-03-25T19:19:48.234 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:48.234 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:48.234 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:48.234 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:48.234 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:48.234 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:48.234 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:48.234 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:48.234 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:48.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.233+0000 7f67adcd3640 5 monclient: authenticate success, global_id 4129 2024-03-25T19:19:48.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.233+0000 7f67adcd3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:19:48.236 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.236+0000 7f67a6ffd640 1 -- 172.21.15.103:0/4218864120 <== mon.1 v1:172.21.15.103:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f67900017d0 con 0x7f67a8142a20 2024-03-25T19:19:48.236 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.236+0000 7f67a6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:19:48.236 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.236+0000 7f67a6ffd640 1 -- 172.21.15.103:0/4218864120 <== mon.1 v1:172.21.15.103:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f6790001be0 con 0x7f67a8142a20 2024-03-25T19:19:48.236 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.236+0000 7f67a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:48.236 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.236+0000 7f67a6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:19:48.236 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.236+0000 7f67a6ffd640 10 monclient: dump: 2024-03-25T19:19:48.236 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 1 2024-03-25T19:19:48.236 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:48.236 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:48.236 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:48.236 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:48.236 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:48.236 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:48.236 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:48.237 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:48.237 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:48.237 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.236+0000 7f67adcd3640 10 monclient: get_monmap_and_config success 2024-03-25T19:19:48.237 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.236+0000 7f67adcd3640 4 set_mon_vals no callback set 2024-03-25T19:19:48.239 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.237+0000 7f67adcd3640 10 monclient: shutdown 2024-03-25T19:19:48.239 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.237+0000 7f67adcd3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:19:48.239 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.237+0000 7f67adcd3640 1 -- 172.21.15.103:0/4218864120 >> v1:172.21.15.103:6789/0 conn(0x7f67a8142a20 legacy=0x7f67a813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:48.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.237+0000 7f67adcd3640 1 -- 172.21.15.103:0/4218864120 shutdown_connections 2024-03-25T19:19:48.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.237+0000 7f67adcd3640 1 -- 172.21.15.103:0/4218864120 >> 172.21.15.103:0/4218864120 conn(0x7f67a80668e0 msgr2=0x7f67a8066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:19:48.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.237+0000 7f67adcd3640 1 -- 172.21.15.103:0/4218864120 shutdown_connections 2024-03-25T19:19:48.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.237+0000 7f67adcd3640 1 -- 172.21.15.103:0/4218864120 wait complete. 2024-03-25T19:19:48.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.237+0000 7f67adcd3640 10 monclient: build_initial_monmap 2024-03-25T19:19:48.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.237+0000 7f67adcd3640 10 monclient: monmap: 2024-03-25T19:19:48.240 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 0 2024-03-25T19:19:48.240 INFO:teuthology.orchestra.run.smithi103.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:19:48.240 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 0.000000 2024-03-25T19:19:48.240 INFO:teuthology.orchestra.run.smithi103.stderr:created 0.000000 2024-03-25T19:19:48.240 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 0 (unknown) 2024-03-25T19:19:48.240 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 1 2024-03-25T19:19:48.240 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:19:48.240 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:19:48.240 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:19:48.240 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:48.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.238+0000 7f67adcd3640 1 librados: starting msgr at 2024-03-25T19:19:48.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.238+0000 7f67adcd3640 1 librados: starting objecter 2024-03-25T19:19:48.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.238+0000 7f67adcd3640 1 Processor -- start 2024-03-25T19:19:48.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.238+0000 7f67adcd3640 1 -- start start 2024-03-25T19:19:48.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.238+0000 7f67adcd3640 1 librados: setting wanted keys 2024-03-25T19:19:48.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.238+0000 7f67adcd3640 1 librados: calling monclient init 2024-03-25T19:19:48.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.238+0000 7f67adcd3640 10 monclient: init 2024-03-25T19:19:48.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.239+0000 7f67adcd3640 10 monclient: _reopen_session rank -1 2024-03-25T19:19:48.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.239+0000 7f67adcd3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:19:48.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.239+0000 7f67adcd3640 10 monclient(hunting): picked mon.noname-c con 0x7f67a80a96b0 addr v1:172.21.15.83:6790/0 2024-03-25T19:19:48.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.239+0000 7f67adcd3640 10 monclient(hunting): picked mon.noname-a con 0x7f67a80b4700 addr v1:172.21.15.83:6789/0 2024-03-25T19:19:48.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.239+0000 7f67adcd3640 10 monclient(hunting): picked mon.noname-b con 0x7f67a8142a20 addr v1:172.21.15.103:6789/0 2024-03-25T19:19:48.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.239+0000 7f67adcd3640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67a81dbed0 con 0x7f67a80b4700 2024-03-25T19:19:48.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.239+0000 7f67adcd3640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67a813a160 con 0x7f67a8142a20 2024-03-25T19:19:48.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.239+0000 7f67adcd3640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67a813a440 con 0x7f67a80a96b0 2024-03-25T19:19:48.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.239+0000 7f67adcd3640 10 monclient(hunting): _renew_subs 2024-03-25T19:19:48.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.239+0000 7f67adcd3640 10 monclient(hunting): authenticate will time out at 6025.373794s 2024-03-25T19:19:48.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.239+0000 7f67a77fe640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f67a80a96b0 0x7f67a813ab40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.103:40446/0 (socket says 172.21.15.103:40446) 2024-03-25T19:19:48.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.239+0000 7f67a77fe640 1 -- 172.21.15.103:0/3055545734 learned_addr learned my addr 172.21.15.103:0/3055545734 (peer_addr_for_me v1:172.21.15.103:0/0) 2024-03-25T19:19:48.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.239+0000 7f67a4ff9640 10 client.?.objecter ms_handle_connect 0x7f67a80a96b0 2024-03-25T19:19:48.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.239+0000 7f67a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:48.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.239+0000 7f67a4ff9640 10 client.?.objecter ms_handle_connect 0x7f67a80b4700 2024-03-25T19:19:48.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.239+0000 7f67a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:48.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.240+0000 7f67a4ff9640 10 client.?.objecter ms_handle_connect 0x7f67a8142a20 2024-03-25T19:19:48.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.240+0000 7f67a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:48.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.241+0000 7f67a4ff9640 1 -- 172.21.15.103:0/3055545734 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3910313207 0 0) 0x7f67a81dbed0 con 0x7f67a80b4700 2024-03-25T19:19:48.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.241+0000 7f67a4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:19:48.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.241+0000 7f67a4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:48.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.241+0000 7f67a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:48.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.241+0000 7f67a4ff9640 10 monclient(hunting): my global_id is 4131 2024-03-25T19:19:48.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.241+0000 7f67a4ff9640 1 -- 172.21.15.103:0/3055545734 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6788004eb0 con 0x7f67a80b4700 2024-03-25T19:19:48.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.241+0000 7f67a4ff9640 1 -- 172.21.15.103:0/3055545734 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4010554918 0 0) 0x7f67a813a440 con 0x7f67a80a96b0 2024-03-25T19:19:48.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.241+0000 7f67a4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:19:48.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.241+0000 7f67a4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:48.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.241+0000 7f67a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:48.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.241+0000 7f67a4ff9640 10 monclient(hunting): my global_id is 4133 2024-03-25T19:19:48.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.241+0000 7f67a4ff9640 1 -- 172.21.15.103:0/3055545734 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67a81dbed0 con 0x7f67a80a96b0 2024-03-25T19:19:48.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.242+0000 7f67a4ff9640 1 -- 172.21.15.103:0/3055545734 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 288048456 0 0) 0x7f67a813a160 con 0x7f67a8142a20 2024-03-25T19:19:48.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.242+0000 7f67a4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:19:48.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.242+0000 7f67a4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:48.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.242+0000 7f67a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:48.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.242+0000 7f67a4ff9640 10 monclient(hunting): my global_id is 4132 2024-03-25T19:19:48.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.242+0000 7f67a4ff9640 1 -- 172.21.15.103:0/3055545734 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67a813a440 con 0x7f67a8142a20 2024-03-25T19:19:48.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.246+0000 7f67a4ff9640 1 -- 172.21.15.103:0/3055545734 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1786208397 0 0) 0x7f67a81dbed0 con 0x7f67a80a96b0 2024-03-25T19:19:48.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.246+0000 7f67a4ff9640 1 -- 172.21.15.103:0/3055545734 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67a813a160 con 0x7f67a80a96b0 2024-03-25T19:19:48.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.246+0000 7f67a4ff9640 1 -- 172.21.15.103:0/3055545734 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 688973613 0 0) 0x7f6788004eb0 con 0x7f67a80b4700 2024-03-25T19:19:48.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.246+0000 7f67a4ff9640 1 -- 172.21.15.103:0/3055545734 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67a81dbed0 con 0x7f67a80b4700 2024-03-25T19:19:48.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.246+0000 7f67a4ff9640 1 -- 172.21.15.103:0/3055545734 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f6790001ae0 con 0x7f67a80b4700 2024-03-25T19:19:48.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.246+0000 7f67a4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:48.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.246+0000 7f67a4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:19:48.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.246+0000 7f67a4ff9640 10 monclient(hunting): dump: 2024-03-25T19:19:48.246 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 1 2024-03-25T19:19:48.247 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:48.247 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:48.247 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:48.247 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:48.247 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:48.247 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:48.247 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:48.247 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:48.247 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:48.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.246+0000 7f67a4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:19:48.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.246+0000 7f67a4ff9640 1 -- 172.21.15.103:0/3055545734 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f67980019f0 con 0x7f67a80a96b0 2024-03-25T19:19:48.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.246+0000 7f67a4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:48.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.246+0000 7f67a4ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:19:48.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.246+0000 7f67a4ff9640 10 monclient(hunting): dump: 2024-03-25T19:19:48.247 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 1 2024-03-25T19:19:48.247 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:48.248 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:48.248 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:48.248 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:48.248 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:48.248 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:48.248 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:48.248 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:48.248 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:48.249 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.247+0000 7f67a4ff9640 1 -- 172.21.15.103:0/3055545734 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 865477055 0 0) 0x7f67a813a440 con 0x7f67a8142a20 2024-03-25T19:19:48.249 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.247+0000 7f67a4ff9640 1 -- 172.21.15.103:0/3055545734 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6788004eb0 con 0x7f67a8142a20 2024-03-25T19:19:48.249 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.247+0000 7f67a4ff9640 1 -- 172.21.15.103:0/3055545734 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f67a0001a30 con 0x7f67a8142a20 2024-03-25T19:19:48.249 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.247+0000 7f67a4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:48.249 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.247+0000 7f67a4ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:19:48.249 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.247+0000 7f67a4ff9640 10 monclient(hunting): dump: 2024-03-25T19:19:48.249 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 1 2024-03-25T19:19:48.250 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:48.250 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:48.250 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:48.250 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:48.250 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:48.250 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:48.250 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:48.250 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:48.250 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:48.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.249+0000 7f67a4ff9640 1 -- 172.21.15.103:0/3055545734 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 767632649 0 0) 0x7f67a81dbed0 con 0x7f67a80b4700 2024-03-25T19:19:48.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.249+0000 7f67a4ff9640 1 -- 172.21.15.103:0/3055545734 >> v1:172.21.15.83:6790/0 conn(0x7f67a80a96b0 legacy=0x7f67a813ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:48.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.249+0000 7f67a4ff9640 1 -- 172.21.15.103:0/3055545734 >> v1:172.21.15.103:6789/0 conn(0x7f67a8142a20 legacy=0x7f67a81db7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:48.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.249+0000 7f67a4ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:19:48.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.249+0000 7f67a4ff9640 1 monclient: found mon.a 2024-03-25T19:19:48.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.249+0000 7f67a4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:19:48.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.250+0000 7f67a4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:19:48.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.250+0000 7f67a4ff9640 1 -- 172.21.15.103:0/3055545734 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67a813a720 con 0x7f67a80b4700 2024-03-25T19:19:48.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.250+0000 7f67a4ff9640 10 monclient: _finish_auth 0 2024-03-25T19:19:48.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.250+0000 7f67a4ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:19:48.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.250+0000 7f67a4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:19:48.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.250+0000 7f67adcd3640 5 monclient: authenticate success, global_id 4131 2024-03-25T19:19:48.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.250+0000 7f67adcd3640 10 monclient: _renew_subs 2024-03-25T19:19:48.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.250+0000 7f67adcd3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:19:48.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.250+0000 7f67adcd3640 1 -- 172.21.15.103:0/3055545734 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67a81388c0 con 0x7f67a80b4700 2024-03-25T19:19:48.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.250+0000 7f67adcd3640 10 client.4131.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:19:48.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.250+0000 7f67adcd3640 10 monclient: _renew_subs 2024-03-25T19:19:48.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.250+0000 7f67adcd3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:19:48.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.250+0000 7f67adcd3640 1 -- 172.21.15.103:0/3055545734 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f67a8139130 con 0x7f67a80b4700 2024-03-25T19:19:48.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.250+0000 7f67adcd3640 1 librados: init done 2024-03-25T19:19:48.253 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.250+0000 7f67adcd3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:48.253 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.251+0000 7f67adcd3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:48.253 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.251+0000 7f67adcd3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:19:48.253 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.251+0000 7f67adcd3640 1 -- 172.21.15.103:0/3055545734 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67a804f730 con 0x7f67a80b4700 2024-03-25T19:19:48.253 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.253+0000 7f67a4ff9640 1 -- 172.21.15.103:0/3055545734 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f67900038a0 con 0x7f67a80b4700 2024-03-25T19:19:48.253 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.253+0000 7f67a4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:19:48.253 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.253+0000 7f67a67fc640 4 set_mon_vals no callback set 2024-03-25T19:19:48.253 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.253+0000 7f67a4ff9640 1 -- 172.21.15.103:0/3055545734 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f6790003bb0 con 0x7f67a80b4700 2024-03-25T19:19:48.253 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.253+0000 7f67a4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:48.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.253+0000 7f67a4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:19:48.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.253+0000 7f67a4ff9640 10 monclient: dump: 2024-03-25T19:19:48.254 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 1 2024-03-25T19:19:48.254 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:48.254 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:48.254 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:48.254 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:48.254 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:48.254 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:48.254 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:48.254 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:48.254 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:48.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.253+0000 7f67a4ff9640 1 -- 172.21.15.103:0/3055545734 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f67900040a0 con 0x7f67a80b4700 2024-03-25T19:19:48.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.253+0000 7f67a4ff9640 1 -- 172.21.15.103:0/3055545734 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 3799170078 0 0) 0x7f6790005550 con 0x7f67a80b4700 2024-03-25T19:19:48.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.253+0000 7f67a4ff9640 10 client.4131.objecter ms_dispatch 0x7f67a8000b90 osd_map(4..4 src has 1..4) v4 2024-03-25T19:19:48.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.253+0000 7f67a4ff9640 3 client.4131.objecter handle_osd_map got epochs [4,4] > 0 2024-03-25T19:19:48.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.253+0000 7f67a4ff9640 3 client.4131.objecter handle_osd_map decoding full epoch 4 2024-03-25T19:19:48.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.253+0000 7f67a4ff9640 20 client.4131.objecter dump_active .. 0 homeless 2024-03-25T19:19:48.348 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.349+0000 7f67a4ff9640 1 -- 172.21.15.103:0/3055545734 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f67900051e0 con 0x7f67a80b4700 2024-03-25T19:19:48.348 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.349+0000 7f67a4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:48.348 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.349+0000 7f67a4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:19:48.386 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-03-25T19:19:48.387+0000 7f02a242b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T19:19:48.424 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.424+0000 7f67adcd3640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "467ce07c-8165-4299-90b2-274bc771cff5", "id": 2}] 2024-03-25T19:19:48.424 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.424+0000 7f67adcd3640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "467ce07c-8165-4299-90b2-274bc771cff5", "id": 2}] 2024-03-25T19:19:48.424 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.424+0000 7f67adcd3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:19:48.424 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.424+0000 7f67adcd3640 1 -- 172.21.15.103:0/3055545734 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd new", "uuid": "467ce07c-8165-4299-90b2-274bc771cff5", "id": 2} v 0) v1 -- 0x7f67a8139530 con 0x7f67a80b4700 2024-03-25T19:19:48.455 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.455+0000 7f67a4ff9640 1 -- 172.21.15.103:0/3055545734 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "467ce07c-8165-4299-90b2-274bc771cff5", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 2079806370 0 216599125) 0x7f6790005780 con 0x7f67a80b4700 2024-03-25T19:19:48.455 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.455+0000 7f67a4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "467ce07c-8165-4299-90b2-274bc771cff5", "id": 2}] 2024-03-25T19:19:48.455 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.455+0000 7f67a4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:19:48.455 INFO:teuthology.orchestra.run.smithi103.stdout:2 2024-03-25T19:19:48.456 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.456+0000 7f67adcd3640 10 librados: watch_flush enter 2024-03-25T19:19:48.456 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.456+0000 7f67adcd3640 10 librados: watch_flush exit 2024-03-25T19:19:48.456 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.456+0000 7f67adcd3640 20 client.4131.objecter shutdown clearing up homeless session... 2024-03-25T19:19:48.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.456+0000 7f67adcd3640 10 client.4131.objecter successfully canceled tick 2024-03-25T19:19:48.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.456+0000 7f67adcd3640 10 monclient: shutdown 2024-03-25T19:19:48.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.456+0000 7f67adcd3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:19:48.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.456+0000 7f67adcd3640 1 -- 172.21.15.103:0/3055545734 >> v1:172.21.15.83:6789/0 conn(0x7f67a80b4700 legacy=0x7f67a81db0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:48.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.456+0000 7f67adcd3640 1 -- 172.21.15.103:0/3055545734 shutdown_connections 2024-03-25T19:19:48.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.456+0000 7f67adcd3640 1 -- 172.21.15.103:0/3055545734 >> 172.21.15.103:0/3055545734 conn(0x7f67a80668e0 msgr2=0x7f67a80b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:19:48.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.456+0000 7f67adcd3640 1 -- 172.21.15.103:0/3055545734 shutdown_connections 2024-03-25T19:19:48.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.456+0000 7f67adcd3640 1 -- 172.21.15.103:0/3055545734 wait complete. 2024-03-25T19:19:48.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.456+0000 7f67adcd3640 1 librados: shutdown 2024-03-25T19:19:48.467 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new a412567e-e1ad-4d42-8e3a-344cbe4fb874 3 2024-03-25T19:19:48.490 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-25T19:19:48.491+0000 7f54aa63c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T19:19:48.529 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-03-25T19:19:48.529+0000 7f02a242b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T19:19:48.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.567+0000 7ff416425640 10 monclient: get_monmap_and_config 2024-03-25T19:19:48.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.567+0000 7ff416425640 10 monclient: build_initial_monmap 2024-03-25T19:19:48.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.567+0000 7ff416425640 10 monclient: monmap: 2024-03-25T19:19:48.567 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 0 2024-03-25T19:19:48.567 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:48.567 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:19:48.568359+0000 2024-03-25T19:19:48.567 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:19:48.568359+0000 2024-03-25T19:19:48.567 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 0 (unknown) 2024-03-25T19:19:48.567 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:48.567 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:19:48.568 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:19:48.568 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:19:48.568 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:48.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.568+0000 7ff416425640 1 Processor -- start 2024-03-25T19:19:48.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.568+0000 7ff416425640 1 -- start start 2024-03-25T19:19:48.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.568+0000 7ff416425640 10 monclient: init 2024-03-25T19:19:48.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.569+0000 7ff416425640 10 monclient: _reopen_session rank -1 2024-03-25T19:19:48.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.569+0000 7ff416425640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:19:48.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.569+0000 7ff416425640 10 monclient(hunting): picked mon.noname-b con 0x7ff4100b4700 addr v1:172.21.15.83:6790/0 2024-03-25T19:19:48.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.569+0000 7ff416425640 10 monclient(hunting): picked mon.noname-c con 0x7ff410142a20 addr v1:172.21.15.103:6789/0 2024-03-25T19:19:48.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.569+0000 7ff416425640 10 monclient(hunting): picked mon.noname-a con 0x7ff4100a96b0 addr v1:172.21.15.83:6789/0 2024-03-25T19:19:48.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.569+0000 7ff416425640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4100a9d60 con 0x7ff4100a96b0 2024-03-25T19:19:48.570 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.569+0000 7ff416425640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff41013ff10 con 0x7ff410142a20 2024-03-25T19:19:48.570 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.569+0000 7ff416425640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4100aa1f0 con 0x7ff4100b4700 2024-03-25T19:19:48.570 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.569+0000 7ff416425640 10 monclient(hunting): _renew_subs 2024-03-25T19:19:48.570 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.569+0000 7ff416425640 10 monclient(hunting): authenticate will time out at 6025.703795s 2024-03-25T19:19:48.570 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.569+0000 7ff4149c2640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ff4100a96b0 0x7ff41013f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.103:42330/0 (socket says 172.21.15.103:42330) 2024-03-25T19:19:48.570 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.569+0000 7ff4149c2640 1 -- 172.21.15.103:0/1450664823 learned_addr learned my addr 172.21.15.103:0/1450664823 (peer_addr_for_me v1:172.21.15.103:0/0) 2024-03-25T19:19:48.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.572+0000 7ff40effd640 1 -- 172.21.15.103:0/1450664823 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2925047447 0 0) 0x7ff4100aa1f0 con 0x7ff4100b4700 2024-03-25T19:19:48.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.572+0000 7ff40effd640 10 monclient(hunting): _negotiate 2024-03-25T19:19:48.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.572+0000 7ff40effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:48.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.572+0000 7ff40effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:48.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.572+0000 7ff40effd640 10 monclient(hunting): my global_id is 4136 2024-03-25T19:19:48.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.572+0000 7ff40effd640 1 -- 172.21.15.103:0/1450664823 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3f8003520 con 0x7ff4100b4700 2024-03-25T19:19:48.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.572+0000 7ff40effd640 1 -- 172.21.15.103:0/1450664823 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1418848270 0 0) 0x7ff41013ff10 con 0x7ff410142a20 2024-03-25T19:19:48.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.572+0000 7ff40effd640 10 monclient(hunting): _negotiate 2024-03-25T19:19:48.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.572+0000 7ff40effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:48.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.572+0000 7ff40effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:48.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.572+0000 7ff40effd640 10 monclient(hunting): my global_id is 4135 2024-03-25T19:19:48.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.572+0000 7ff40effd640 1 -- 172.21.15.103:0/1450664823 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4100aa1f0 con 0x7ff410142a20 2024-03-25T19:19:48.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.572+0000 7ff40effd640 1 -- 172.21.15.103:0/1450664823 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4187106663 0 0) 0x7ff4100a9d60 con 0x7ff4100a96b0 2024-03-25T19:19:48.573 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.572+0000 7ff40effd640 10 monclient(hunting): _negotiate 2024-03-25T19:19:48.573 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.572+0000 7ff40effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:48.573 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.572+0000 7ff40effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:48.573 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.572+0000 7ff40effd640 10 monclient(hunting): my global_id is 4134 2024-03-25T19:19:48.573 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.572+0000 7ff40effd640 1 -- 172.21.15.103:0/1450664823 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff41013ff10 con 0x7ff4100a96b0 2024-03-25T19:19:48.575 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.575+0000 7ff40effd640 1 -- 172.21.15.103:0/1450664823 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3620316659 0 0) 0x7ff4100aa1f0 con 0x7ff410142a20 2024-03-25T19:19:48.575 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.576+0000 7ff40effd640 1 -- 172.21.15.103:0/1450664823 >> v1:172.21.15.83:6790/0 conn(0x7ff4100b4700 legacy=0x7ff410142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:48.575 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.576+0000 7ff40effd640 1 -- 172.21.15.103:0/1450664823 >> v1:172.21.15.83:6789/0 conn(0x7ff4100a96b0 legacy=0x7ff41013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:48.575 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.576+0000 7ff40effd640 10 monclient: _finish_hunting 0 2024-03-25T19:19:48.575 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.576+0000 7ff40effd640 1 monclient: found mon.noname-c 2024-03-25T19:19:48.576 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.576+0000 7ff40effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:19:48.576 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.576+0000 7ff40effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.103:6789/0 2024-03-25T19:19:48.576 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.576+0000 7ff40effd640 1 -- 172.21.15.103:0/1450664823 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4100b4ae0 con 0x7ff410142a20 2024-03-25T19:19:48.576 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.576+0000 7ff40effd640 10 monclient: _finish_auth 0 2024-03-25T19:19:48.576 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.576+0000 7ff40effd640 10 monclient: _check_auth_tickets 2024-03-25T19:19:48.576 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.576+0000 7ff40effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:19:48.576 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.576+0000 7ff40effd640 1 -- 172.21.15.103:0/1450664823 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7ff400001670 con 0x7ff410142a20 2024-03-25T19:19:48.576 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.576+0000 7ff40effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:48.576 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.576+0000 7ff40effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:19:48.576 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.576+0000 7ff40effd640 10 monclient: dump: 2024-03-25T19:19:48.576 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 1 2024-03-25T19:19:48.576 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:48.576 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:48.576 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:48.576 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:48.576 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:48.577 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:48.577 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:48.577 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:48.577 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:48.577 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.576+0000 7ff416425640 5 monclient: authenticate success, global_id 4135 2024-03-25T19:19:48.577 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.576+0000 7ff416425640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:19:48.578 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.577+0000 7ff40effd640 1 -- 172.21.15.103:0/1450664823 <== mon.1 v1:172.21.15.103:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4000017d0 con 0x7ff410142a20 2024-03-25T19:19:48.578 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.577+0000 7ff40effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:19:48.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.577+0000 7ff40effd640 1 -- 172.21.15.103:0/1450664823 <== mon.1 v1:172.21.15.103:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7ff400001be0 con 0x7ff410142a20 2024-03-25T19:19:48.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.577+0000 7ff40effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:48.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.577+0000 7ff40effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:19:48.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.577+0000 7ff40effd640 10 monclient: dump: 2024-03-25T19:19:48.579 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 1 2024-03-25T19:19:48.579 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:48.579 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:48.579 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:48.579 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:48.579 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:48.579 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:48.579 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:48.579 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:48.579 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:48.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.577+0000 7ff416425640 10 monclient: get_monmap_and_config success 2024-03-25T19:19:48.580 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.577+0000 7ff416425640 4 set_mon_vals no callback set 2024-03-25T19:19:48.580 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.578+0000 7ff416425640 10 monclient: shutdown 2024-03-25T19:19:48.580 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.578+0000 7ff416425640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:19:48.580 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.578+0000 7ff416425640 1 -- 172.21.15.103:0/1450664823 >> v1:172.21.15.103:6789/0 conn(0x7ff410142a20 legacy=0x7ff41013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:48.580 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.578+0000 7ff416425640 1 -- 172.21.15.103:0/1450664823 shutdown_connections 2024-03-25T19:19:48.580 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.578+0000 7ff416425640 1 -- 172.21.15.103:0/1450664823 >> 172.21.15.103:0/1450664823 conn(0x7ff4100668e0 msgr2=0x7ff410066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:19:48.580 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.578+0000 7ff416425640 1 -- 172.21.15.103:0/1450664823 shutdown_connections 2024-03-25T19:19:48.580 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.578+0000 7ff416425640 1 -- 172.21.15.103:0/1450664823 wait complete. 2024-03-25T19:19:48.580 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.579+0000 7ff416425640 10 monclient: build_initial_monmap 2024-03-25T19:19:48.580 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.579+0000 7ff416425640 10 monclient: monmap: 2024-03-25T19:19:48.580 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 0 2024-03-25T19:19:48.580 INFO:teuthology.orchestra.run.smithi103.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:19:48.580 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 0.000000 2024-03-25T19:19:48.580 INFO:teuthology.orchestra.run.smithi103.stderr:created 0.000000 2024-03-25T19:19:48.581 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 0 (unknown) 2024-03-25T19:19:48.581 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 1 2024-03-25T19:19:48.581 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:19:48.581 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:19:48.581 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:19:48.581 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:48.581 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.579+0000 7ff416425640 1 librados: starting msgr at 2024-03-25T19:19:48.581 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.579+0000 7ff416425640 1 librados: starting objecter 2024-03-25T19:19:48.581 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.579+0000 7ff416425640 1 Processor -- start 2024-03-25T19:19:48.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.580+0000 7ff416425640 1 -- start start 2024-03-25T19:19:48.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.580+0000 7ff416425640 1 librados: setting wanted keys 2024-03-25T19:19:48.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.580+0000 7ff416425640 1 librados: calling monclient init 2024-03-25T19:19:48.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.580+0000 7ff416425640 10 monclient: init 2024-03-25T19:19:48.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.580+0000 7ff416425640 10 monclient: _reopen_session rank -1 2024-03-25T19:19:48.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.580+0000 7ff416425640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:19:48.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.580+0000 7ff416425640 10 monclient(hunting): picked mon.noname-b con 0x7ff4100a96b0 addr v1:172.21.15.103:6789/0 2024-03-25T19:19:48.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.580+0000 7ff416425640 10 monclient(hunting): picked mon.noname-a con 0x7ff4100b4700 addr v1:172.21.15.83:6789/0 2024-03-25T19:19:48.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.580+0000 7ff416425640 10 monclient(hunting): picked mon.noname-c con 0x7ff410142a20 addr v1:172.21.15.83:6790/0 2024-03-25T19:19:48.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.580+0000 7ff416425640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4101daba0 con 0x7ff4100b4700 2024-03-25T19:19:48.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.580+0000 7ff416425640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4100acf70 con 0x7ff4100a96b0 2024-03-25T19:19:48.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.580+0000 7ff416425640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4101db110 con 0x7ff410142a20 2024-03-25T19:19:48.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.580+0000 7ff416425640 10 monclient(hunting): _renew_subs 2024-03-25T19:19:48.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.580+0000 7ff416425640 10 monclient(hunting): authenticate will time out at 6025.715289s 2024-03-25T19:19:48.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.581+0000 7ff4149c2640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7ff410142a20 0x7ff4100ac860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.103:40460/0 (socket says 172.21.15.103:40460) 2024-03-25T19:19:48.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.581+0000 7ff4149c2640 1 -- 172.21.15.103:0/3057721717 learned_addr learned my addr 172.21.15.103:0/3057721717 (peer_addr_for_me v1:172.21.15.103:0/0) 2024-03-25T19:19:48.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.581+0000 7ff40f7fe640 1 --1- 172.21.15.103:0/3057721717 >> v1:172.21.15.83:6789/0 conn(0x7ff4100b4700 0x7ff41013a860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.103:42340/0 (socket says 172.21.15.103:42340) 2024-03-25T19:19:48.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.581+0000 7ff40cff9640 10 client.?.objecter ms_handle_connect 0x7ff4100b4700 2024-03-25T19:19:48.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.581+0000 7ff40cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:48.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.581+0000 7ff40cff9640 10 client.?.objecter ms_handle_connect 0x7ff410142a20 2024-03-25T19:19:48.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.581+0000 7ff40cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:48.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.581+0000 7ff40cff9640 10 client.?.objecter ms_handle_connect 0x7ff4100a96b0 2024-03-25T19:19:48.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.581+0000 7ff40cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:48.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.583+0000 7ff40cff9640 1 -- 172.21.15.103:0/3057721717 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 275358795 0 0) 0x7ff4101daba0 con 0x7ff4100b4700 2024-03-25T19:19:48.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.583+0000 7ff40cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:19:48.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.583+0000 7ff40cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:48.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.583+0000 7ff40cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:48.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.583+0000 7ff40cff9640 10 monclient(hunting): my global_id is 4137 2024-03-25T19:19:48.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.583+0000 7ff40cff9640 1 -- 172.21.15.103:0/3057721717 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3fc004eb0 con 0x7ff4100b4700 2024-03-25T19:19:48.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.583+0000 7ff40cff9640 1 -- 172.21.15.103:0/3057721717 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2851222450 0 0) 0x7ff4101db110 con 0x7ff410142a20 2024-03-25T19:19:48.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.583+0000 7ff40cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:19:48.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.583+0000 7ff40cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:48.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.583+0000 7ff40cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:48.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.583+0000 7ff40cff9640 10 monclient(hunting): my global_id is 4139 2024-03-25T19:19:48.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.583+0000 7ff40cff9640 1 -- 172.21.15.103:0/3057721717 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4101daba0 con 0x7ff410142a20 2024-03-25T19:19:48.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.584+0000 7ff40cff9640 1 -- 172.21.15.103:0/3057721717 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1125561236 0 0) 0x7ff4100acf70 con 0x7ff4100a96b0 2024-03-25T19:19:48.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.584+0000 7ff40cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:19:48.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.584+0000 7ff40cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:48.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.584+0000 7ff40cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:48.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.584+0000 7ff40cff9640 10 monclient(hunting): my global_id is 4138 2024-03-25T19:19:48.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.584+0000 7ff40cff9640 1 -- 172.21.15.103:0/3057721717 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4101db110 con 0x7ff4100a96b0 2024-03-25T19:19:48.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.587+0000 7ff40cff9640 1 -- 172.21.15.103:0/3057721717 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2413249904 0 0) 0x7ff3fc004eb0 con 0x7ff4100b4700 2024-03-25T19:19:48.587 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.587+0000 7ff40cff9640 1 -- 172.21.15.103:0/3057721717 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff4100acf70 con 0x7ff4100b4700 2024-03-25T19:19:48.587 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.587+0000 7ff40cff9640 1 -- 172.21.15.103:0/3057721717 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7ff400001ba0 con 0x7ff4100b4700 2024-03-25T19:19:48.587 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.587+0000 7ff40cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:48.587 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.587+0000 7ff40cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:19:48.587 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.587+0000 7ff40cff9640 10 monclient(hunting): dump: 2024-03-25T19:19:48.587 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 1 2024-03-25T19:19:48.587 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:48.587 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:48.587 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:48.587 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:48.587 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:48.587 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:48.587 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:48.587 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:48.587 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:48.587 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.587+0000 7ff40cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:19:48.588 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.588+0000 7ff40cff9640 1 -- 172.21.15.103:0/3057721717 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1221467808 0 0) 0x7ff4101daba0 con 0x7ff410142a20 2024-03-25T19:19:48.588 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.588+0000 7ff40cff9640 1 -- 172.21.15.103:0/3057721717 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3fc004eb0 con 0x7ff410142a20 2024-03-25T19:19:48.588 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.589+0000 7ff40cff9640 1 -- 172.21.15.103:0/3057721717 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 910507414 0 0) 0x7ff4101db110 con 0x7ff4100a96b0 2024-03-25T19:19:48.588 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.589+0000 7ff40cff9640 1 -- 172.21.15.103:0/3057721717 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff4101daba0 con 0x7ff4100a96b0 2024-03-25T19:19:48.588 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.589+0000 7ff40cff9640 1 -- 172.21.15.103:0/3057721717 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7ff404001880 con 0x7ff4100a96b0 2024-03-25T19:19:48.588 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.589+0000 7ff40cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:48.589 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.589+0000 7ff40cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:19:48.589 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.589+0000 7ff40cff9640 10 monclient(hunting): dump: 2024-03-25T19:19:48.589 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 1 2024-03-25T19:19:48.589 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:48.589 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:48.589 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:48.589 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:48.589 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:48.589 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:48.589 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:48.589 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:48.589 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:48.591 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.591+0000 7ff40cff9640 1 -- 172.21.15.103:0/3057721717 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7ff408001bb0 con 0x7ff410142a20 2024-03-25T19:19:48.591 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.591+0000 7ff40cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:48.591 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.591+0000 7ff40cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:19:48.591 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.591+0000 7ff40cff9640 10 monclient(hunting): dump: 2024-03-25T19:19:48.591 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 1 2024-03-25T19:19:48.592 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:48.592 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:48.592 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:48.592 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:48.592 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:48.592 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:48.592 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:48.592 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:48.592 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:48.592 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.591+0000 7ff40cff9640 1 -- 172.21.15.103:0/3057721717 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2511493612 0 0) 0x7ff3fc004eb0 con 0x7ff410142a20 2024-03-25T19:19:48.592 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.591+0000 7ff40cff9640 1 -- 172.21.15.103:0/3057721717 >> v1:172.21.15.103:6789/0 conn(0x7ff4100a96b0 legacy=0x7ff4100abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:48.592 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.591+0000 7ff40cff9640 1 -- 172.21.15.103:0/3057721717 >> v1:172.21.15.83:6789/0 conn(0x7ff4100b4700 legacy=0x7ff41013a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:48.592 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.591+0000 7ff40cff9640 10 monclient: _finish_hunting 0 2024-03-25T19:19:48.592 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.591+0000 7ff40cff9640 1 monclient: found mon.c 2024-03-25T19:19:48.592 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.591+0000 7ff40cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:19:48.592 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.591+0000 7ff40cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:19:48.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.591+0000 7ff40cff9640 1 -- 172.21.15.103:0/3057721717 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4101db2a0 con 0x7ff410142a20 2024-03-25T19:19:48.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.591+0000 7ff40cff9640 10 monclient: _finish_auth 0 2024-03-25T19:19:48.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.591+0000 7ff40cff9640 10 monclient: _check_auth_tickets 2024-03-25T19:19:48.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.591+0000 7ff40cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:19:48.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.591+0000 7ff416425640 5 monclient: authenticate success, global_id 4139 2024-03-25T19:19:48.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.591+0000 7ff416425640 10 monclient: _renew_subs 2024-03-25T19:19:48.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.591+0000 7ff416425640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:19:48.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.591+0000 7ff416425640 1 -- 172.21.15.103:0/3057721717 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4101db4f0 con 0x7ff410142a20 2024-03-25T19:19:48.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.591+0000 7ff416425640 10 client.4139.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:19:48.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.591+0000 7ff416425640 10 monclient: _renew_subs 2024-03-25T19:19:48.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.591+0000 7ff416425640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:19:48.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.591+0000 7ff416425640 1 -- 172.21.15.103:0/3057721717 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4101362d0 con 0x7ff410142a20 2024-03-25T19:19:48.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.591+0000 7ff416425640 1 librados: init done 2024-03-25T19:19:48.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.592+0000 7ff416425640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:48.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.592+0000 7ff416425640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:48.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.592+0000 7ff416425640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:19:48.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.592+0000 7ff416425640 1 -- 172.21.15.103:0/3057721717 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4100b1dc0 con 0x7ff410142a20 2024-03-25T19:19:48.625 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-25T19:19:48.626+0000 7f54aa63c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T19:19:48.673 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-03-25T19:19:48.674+0000 7f02a242b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T19:19:48.719 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-03-25T19:19:48.719+0000 7f02a242b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T19:19:48.801 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.801+0000 7ff40cff9640 1 -- 172.21.15.103:0/3057721717 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff408002080 con 0x7ff410142a20 2024-03-25T19:19:48.801 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.801+0000 7ff40cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:19:48.801 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.801+0000 7ff40cff9640 1 -- 172.21.15.103:0/3057721717 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7ff408004d00 con 0x7ff410142a20 2024-03-25T19:19:48.801 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.801+0000 7ff40cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:48.801 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.801+0000 7ff40cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:19:48.801 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.801+0000 7ff40cff9640 10 monclient: dump: 2024-03-25T19:19:48.801 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 1 2024-03-25T19:19:48.801 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:48.801 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:48.802 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:48.802 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:48.802 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:48.802 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:48.802 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:48.802 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:48.802 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:48.802 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.801+0000 7ff40cff9640 1 -- 172.21.15.103:0/3057721717 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7ff4080052b0 con 0x7ff410142a20 2024-03-25T19:19:48.802 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.801+0000 7ff40e7fc640 4 set_mon_vals no callback set 2024-03-25T19:19:48.803 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.804+0000 7ff40cff9640 1 -- 172.21.15.103:0/3057721717 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 1170735281 0 0) 0x7ff408006980 con 0x7ff410142a20 2024-03-25T19:19:48.804 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.804+0000 7ff40cff9640 10 client.4139.objecter ms_dispatch 0x7ff410000b90 osd_map(5..5 src has 1..5) v4 2024-03-25T19:19:48.804 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.804+0000 7ff40cff9640 3 client.4139.objecter handle_osd_map got epochs [5,5] > 0 2024-03-25T19:19:48.804 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.804+0000 7ff40cff9640 3 client.4139.objecter handle_osd_map decoding full epoch 5 2024-03-25T19:19:48.804 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.804+0000 7ff40cff9640 20 client.4139.objecter dump_active .. 0 homeless 2024-03-25T19:19:48.804 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.804+0000 7ff40cff9640 1 -- 172.21.15.103:0/3057721717 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ff4080057e0 con 0x7ff410142a20 2024-03-25T19:19:48.804 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.804+0000 7ff40cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:48.804 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.804+0000 7ff40cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:19:48.832 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-03-25T19:19:48.832+0000 7f02a242b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:19:48.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.876+0000 7ff416425640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "a412567e-e1ad-4d42-8e3a-344cbe4fb874", "id": 3}] 2024-03-25T19:19:48.876 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.876+0000 7ff416425640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "a412567e-e1ad-4d42-8e3a-344cbe4fb874", "id": 3}] 2024-03-25T19:19:48.876 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.876+0000 7ff416425640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:19:48.876 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.876+0000 7ff416425640 1 -- 172.21.15.103:0/3057721717 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd new", "uuid": "a412567e-e1ad-4d42-8e3a-344cbe4fb874", "id": 3} v 0) v1 -- 0x7ff4101dad90 con 0x7ff410142a20 2024-03-25T19:19:48.931 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-03-25T19:19:48.931+0000 7f02a242b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T19:19:48.981 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.982+0000 7ff40cff9640 1 -- 172.21.15.103:0/3057721717 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a412567e-e1ad-4d42-8e3a-344cbe4fb874", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 2076874195 0 525046306) 0x7ff4080064f0 con 0x7ff410142a20 2024-03-25T19:19:48.981 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.982+0000 7ff40cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "a412567e-e1ad-4d42-8e3a-344cbe4fb874", "id": 3}] 2024-03-25T19:19:48.981 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.982+0000 7ff40cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:19:48.982 INFO:teuthology.orchestra.run.smithi103.stdout:3 2024-03-25T19:19:48.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.983+0000 7ff416425640 10 librados: watch_flush enter 2024-03-25T19:19:48.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.983+0000 7ff416425640 10 librados: watch_flush exit 2024-03-25T19:19:48.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.983+0000 7ff416425640 20 client.4139.objecter shutdown clearing up homeless session... 2024-03-25T19:19:48.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.983+0000 7ff416425640 10 client.4139.objecter successfully canceled tick 2024-03-25T19:19:48.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.983+0000 7ff416425640 10 monclient: shutdown 2024-03-25T19:19:48.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.983+0000 7ff416425640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:19:48.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.983+0000 7ff416425640 1 -- 172.21.15.103:0/3057721717 >> v1:172.21.15.83:6790/0 conn(0x7ff410142a20 legacy=0x7ff4100ac860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:48.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.983+0000 7ff416425640 1 -- 172.21.15.103:0/3057721717 shutdown_connections 2024-03-25T19:19:48.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.983+0000 7ff416425640 1 -- 172.21.15.103:0/3057721717 >> 172.21.15.103:0/3057721717 conn(0x7ff4100668e0 msgr2=0x7ff41013f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:19:48.984 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.983+0000 7ff416425640 1 -- 172.21.15.103:0/3057721717 shutdown_connections 2024-03-25T19:19:48.984 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.983+0000 7ff416425640 1 -- 172.21.15.103:0/3057721717 wait complete. 2024-03-25T19:19:48.984 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:48.983+0000 7ff416425640 1 librados: shutdown 2024-03-25T19:19:48.995 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new 8354d57d-5c88-4259-995b-94c71f146960 4 2024-03-25T19:19:49.024 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-25T19:19:49.025+0000 7f54aa63c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T19:19:49.070 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-25T19:19:49.070+0000 7f54aa63c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T19:19:49.095 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.095+0000 7f5069f82640 10 monclient: get_monmap_and_config 2024-03-25T19:19:49.095 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.095+0000 7f5069f82640 10 monclient: build_initial_monmap 2024-03-25T19:19:49.095 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.095+0000 7f5069f82640 10 monclient: monmap: 2024-03-25T19:19:49.095 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 0 2024-03-25T19:19:49.095 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:49.095 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:19:49.096397+0000 2024-03-25T19:19:49.095 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:19:49.096397+0000 2024-03-25T19:19:49.095 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 0 (unknown) 2024-03-25T19:19:49.095 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:49.096 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:19:49.096 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:19:49.096 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:19:49.096 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:49.096 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.096+0000 7f5069f82640 1 Processor -- start 2024-03-25T19:19:49.096 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.096+0000 7f5069f82640 1 -- start start 2024-03-25T19:19:49.096 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.096+0000 7f5069f82640 10 monclient: init 2024-03-25T19:19:49.097 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-03-25T19:19:49.097+0000 7f02a242b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:19:49.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.097+0000 7f5069f82640 10 monclient: _reopen_session rank -1 2024-03-25T19:19:49.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.097+0000 7f5069f82640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:19:49.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.097+0000 7f5069f82640 10 monclient(hunting): picked mon.noname-a con 0x7f50640b4700 addr v1:172.21.15.83:6789/0 2024-03-25T19:19:49.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.097+0000 7f5069f82640 10 monclient(hunting): picked mon.noname-c con 0x7f5064142a20 addr v1:172.21.15.103:6789/0 2024-03-25T19:19:49.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.097+0000 7f5069f82640 10 monclient(hunting): picked mon.noname-b con 0x7f50640a96b0 addr v1:172.21.15.83:6790/0 2024-03-25T19:19:49.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.097+0000 7f5069f82640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50640a9d60 con 0x7f50640b4700 2024-03-25T19:19:49.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.097+0000 7f5069f82640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f506413ff10 con 0x7f5064142a20 2024-03-25T19:19:49.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.097+0000 7f5069f82640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50640aa1f0 con 0x7f50640a96b0 2024-03-25T19:19:49.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.097+0000 7f5069f82640 10 monclient(hunting): _renew_subs 2024-03-25T19:19:49.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.097+0000 7f5069f82640 10 monclient(hunting): authenticate will time out at 6026.232145s 2024-03-25T19:19:49.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.098+0000 7f50637fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f50640b4700 0x7f5064142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.103:42344/0 (socket says 172.21.15.103:42344) 2024-03-25T19:19:49.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.098+0000 7f50637fe640 1 -- 172.21.15.103:0/2445789938 learned_addr learned my addr 172.21.15.103:0/2445789938 (peer_addr_for_me v1:172.21.15.103:0/0) 2024-03-25T19:19:49.099 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.098+0000 7f5063fff640 1 --1- 172.21.15.103:0/2445789938 >> v1:172.21.15.83:6790/0 conn(0x7f50640a96b0 0x7f506413f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.103:40472/0 (socket says 172.21.15.103:40472) 2024-03-25T19:19:49.100 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.100+0000 7f5062ffd640 1 -- 172.21.15.103:0/2445789938 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 624320968 0 0) 0x7f50640aa1f0 con 0x7f50640a96b0 2024-03-25T19:19:49.100 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.100+0000 7f5062ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:19:49.100 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.100+0000 7f5062ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:49.100 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.100+0000 7f5062ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:49.100 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.100+0000 7f5062ffd640 10 monclient(hunting): my global_id is 4142 2024-03-25T19:19:49.100 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.100+0000 7f5062ffd640 1 -- 172.21.15.103:0/2445789938 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5050003520 con 0x7f50640a96b0 2024-03-25T19:19:49.100 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.100+0000 7f5062ffd640 1 -- 172.21.15.103:0/2445789938 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2766647008 0 0) 0x7f50640a9d60 con 0x7f50640b4700 2024-03-25T19:19:49.100 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.100+0000 7f5062ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:19:49.100 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.100+0000 7f5062ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:49.100 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.100+0000 7f5062ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:49.100 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.100+0000 7f5062ffd640 10 monclient(hunting): my global_id is 4140 2024-03-25T19:19:49.101 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.100+0000 7f5062ffd640 1 -- 172.21.15.103:0/2445789938 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50640aa1f0 con 0x7f50640b4700 2024-03-25T19:19:49.101 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.100+0000 7f5062ffd640 1 -- 172.21.15.103:0/2445789938 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3938804001 0 0) 0x7f506413ff10 con 0x7f5064142a20 2024-03-25T19:19:49.101 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.100+0000 7f5062ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:19:49.101 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.100+0000 7f5062ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:49.101 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.100+0000 7f5062ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:49.101 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.100+0000 7f5062ffd640 10 monclient(hunting): my global_id is 4141 2024-03-25T19:19:49.101 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.100+0000 7f5062ffd640 1 -- 172.21.15.103:0/2445789938 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50640a9d60 con 0x7f5064142a20 2024-03-25T19:19:49.102 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.103+0000 7f5062ffd640 1 -- 172.21.15.103:0/2445789938 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1508148154 0 0) 0x7f5050003520 con 0x7f50640a96b0 2024-03-25T19:19:49.103 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.103+0000 7f5062ffd640 1 -- 172.21.15.103:0/2445789938 >> v1:172.21.15.103:6789/0 conn(0x7f5064142a20 legacy=0x7f506413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:49.103 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.103+0000 7f5062ffd640 1 -- 172.21.15.103:0/2445789938 >> v1:172.21.15.83:6789/0 conn(0x7f50640b4700 legacy=0x7f5064142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:49.103 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.103+0000 7f5062ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:19:49.103 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.103+0000 7f5062ffd640 1 monclient: found mon.noname-b 2024-03-25T19:19:49.103 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.103+0000 7f5062ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:19:49.103 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.103+0000 7f5062ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.83:6790/0 2024-03-25T19:19:49.103 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.103+0000 7f5062ffd640 1 -- 172.21.15.103:0/2445789938 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50640b4ae0 con 0x7f50640a96b0 2024-03-25T19:19:49.103 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.103+0000 7f5062ffd640 10 monclient: _finish_auth 0 2024-03-25T19:19:49.103 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.103+0000 7f5062ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:19:49.103 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.103+0000 7f5062ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:19:49.103 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.103+0000 7f5069f82640 5 monclient: authenticate success, global_id 4142 2024-03-25T19:19:49.103 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.103+0000 7f5069f82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:19:49.105 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.105+0000 7f5062ffd640 1 -- 172.21.15.103:0/2445789938 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f505c0019d0 con 0x7f50640a96b0 2024-03-25T19:19:49.105 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.105+0000 7f5062ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:49.105 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.106+0000 7f5062ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:19:49.105 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.106+0000 7f5062ffd640 10 monclient: dump: 2024-03-25T19:19:49.105 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 1 2024-03-25T19:19:49.105 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:49.105 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:49.105 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:49.105 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:49.105 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:49.106 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:49.106 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:49.106 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:49.106 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:49.106 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.106+0000 7f5062ffd640 1 -- 172.21.15.103:0/2445789938 <== mon.2 v1:172.21.15.83:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f505c001c20 con 0x7f50640a96b0 2024-03-25T19:19:49.106 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.106+0000 7f5069f82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:19:49.106 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.106+0000 7f5062ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:19:49.106 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.106+0000 7f5069f82640 10 monclient: get_monmap_and_config success 2024-03-25T19:19:49.106 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.106+0000 7f5062ffd640 1 -- 172.21.15.103:0/2445789938 <== mon.2 v1:172.21.15.83:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f505c001fb0 con 0x7f50640a96b0 2024-03-25T19:19:49.106 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.106+0000 7f5062ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:49.106 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.106+0000 7f5069f82640 4 set_mon_vals no callback set 2024-03-25T19:19:49.106 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.106+0000 7f5062ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:19:49.106 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.106+0000 7f5062ffd640 10 monclient: dump: 2024-03-25T19:19:49.107 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 1 2024-03-25T19:19:49.107 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:49.107 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:49.107 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:49.107 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:49.107 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:49.107 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:49.107 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:49.107 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:49.107 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:49.110 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.106+0000 7f5069f82640 10 monclient: shutdown 2024-03-25T19:19:49.110 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.106+0000 7f5069f82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:19:49.110 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.106+0000 7f5069f82640 1 -- 172.21.15.103:0/2445789938 >> v1:172.21.15.83:6790/0 conn(0x7f50640a96b0 legacy=0x7f506413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:49.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.106+0000 7f5069f82640 1 -- 172.21.15.103:0/2445789938 shutdown_connections 2024-03-25T19:19:49.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.106+0000 7f5069f82640 1 -- 172.21.15.103:0/2445789938 >> 172.21.15.103:0/2445789938 conn(0x7f50640668e0 msgr2=0x7f5064066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:19:49.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.106+0000 7f5069f82640 1 -- 172.21.15.103:0/2445789938 shutdown_connections 2024-03-25T19:19:49.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.106+0000 7f5069f82640 1 -- 172.21.15.103:0/2445789938 wait complete. 2024-03-25T19:19:49.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.107+0000 7f5069f82640 10 monclient: build_initial_monmap 2024-03-25T19:19:49.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.107+0000 7f5069f82640 10 monclient: monmap: 2024-03-25T19:19:49.111 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 0 2024-03-25T19:19:49.111 INFO:teuthology.orchestra.run.smithi103.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:19:49.111 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 0.000000 2024-03-25T19:19:49.111 INFO:teuthology.orchestra.run.smithi103.stderr:created 0.000000 2024-03-25T19:19:49.111 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 0 (unknown) 2024-03-25T19:19:49.111 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 1 2024-03-25T19:19:49.111 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:19:49.111 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:19:49.112 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:19:49.112 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:49.112 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.107+0000 7f5069f82640 1 librados: starting msgr at 2024-03-25T19:19:49.112 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.107+0000 7f5069f82640 1 librados: starting objecter 2024-03-25T19:19:49.112 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.107+0000 7f5069f82640 1 Processor -- start 2024-03-25T19:19:49.112 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.107+0000 7f5069f82640 1 -- start start 2024-03-25T19:19:49.112 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.107+0000 7f5069f82640 1 librados: setting wanted keys 2024-03-25T19:19:49.112 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.107+0000 7f5069f82640 1 librados: calling monclient init 2024-03-25T19:19:49.112 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.107+0000 7f5069f82640 10 monclient: init 2024-03-25T19:19:49.112 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.108+0000 7f5069f82640 10 monclient: _reopen_session rank -1 2024-03-25T19:19:49.112 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.108+0000 7f5069f82640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:19:49.112 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.108+0000 7f5069f82640 10 monclient(hunting): picked mon.noname-b con 0x7f50640a96b0 addr v1:172.21.15.103:6789/0 2024-03-25T19:19:49.112 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.108+0000 7f5069f82640 10 monclient(hunting): picked mon.noname-a con 0x7f50640b4700 addr v1:172.21.15.83:6789/0 2024-03-25T19:19:49.112 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.108+0000 7f5069f82640 10 monclient(hunting): picked mon.noname-c con 0x7f5064142a20 addr v1:172.21.15.83:6790/0 2024-03-25T19:19:49.112 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.108+0000 7f5069f82640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f506413c1f0 con 0x7f50640b4700 2024-03-25T19:19:49.112 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.108+0000 7f5069f82640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5064133a00 con 0x7f50640a96b0 2024-03-25T19:19:49.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.108+0000 7f5069f82640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50640ac0e0 con 0x7f5064142a20 2024-03-25T19:19:49.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.108+0000 7f5069f82640 10 monclient(hunting): _renew_subs 2024-03-25T19:19:49.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.108+0000 7f5069f82640 10 monclient(hunting): authenticate will time out at 6026.243117s 2024-03-25T19:19:49.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.108+0000 7f5063fff640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f5064142a20 0x7f50641332f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.103:40484/0 (socket says 172.21.15.103:40484) 2024-03-25T19:19:49.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.108+0000 7f5063fff640 1 -- 172.21.15.103:0/1744190091 learned_addr learned my addr 172.21.15.103:0/1744190091 (peer_addr_for_me v1:172.21.15.103:0/0) 2024-03-25T19:19:49.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.109+0000 7f5060ff9640 10 client.?.objecter ms_handle_connect 0x7f5064142a20 2024-03-25T19:19:49.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.109+0000 7f5060ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:49.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.109+0000 7f5060ff9640 10 client.?.objecter ms_handle_connect 0x7f50640b4700 2024-03-25T19:19:49.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.109+0000 7f5060ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:49.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.109+0000 7f5060ff9640 10 client.?.objecter ms_handle_connect 0x7f50640a96b0 2024-03-25T19:19:49.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.109+0000 7f5060ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:49.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.110+0000 7f5060ff9640 1 -- 172.21.15.103:0/1744190091 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2029394859 0 0) 0x7f50640ac0e0 con 0x7f5064142a20 2024-03-25T19:19:49.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.111+0000 7f5060ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:19:49.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.111+0000 7f5060ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:49.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.111+0000 7f5060ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:49.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.111+0000 7f5060ff9640 10 monclient(hunting): my global_id is 4145 2024-03-25T19:19:49.114 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.111+0000 7f5060ff9640 1 -- 172.21.15.103:0/1744190091 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5044004eb0 con 0x7f5064142a20 2024-03-25T19:19:49.114 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.111+0000 7f5060ff9640 1 -- 172.21.15.103:0/1744190091 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 985045695 0 0) 0x7f506413c1f0 con 0x7f50640b4700 2024-03-25T19:19:49.114 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.111+0000 7f5060ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:19:49.114 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.111+0000 7f5060ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:49.114 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.111+0000 7f5060ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:49.114 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.111+0000 7f5060ff9640 10 monclient(hunting): my global_id is 4143 2024-03-25T19:19:49.114 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.111+0000 7f5060ff9640 1 -- 172.21.15.103:0/1744190091 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50640ac0e0 con 0x7f50640b4700 2024-03-25T19:19:49.114 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.111+0000 7f5060ff9640 1 -- 172.21.15.103:0/1744190091 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2048715786 0 0) 0x7f5064133a00 con 0x7f50640a96b0 2024-03-25T19:19:49.114 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.111+0000 7f5060ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:19:49.114 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.111+0000 7f5060ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:49.114 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.111+0000 7f5060ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:49.114 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.111+0000 7f5060ff9640 10 monclient(hunting): my global_id is 4144 2024-03-25T19:19:49.114 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.111+0000 7f5060ff9640 1 -- 172.21.15.103:0/1744190091 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f506413c1f0 con 0x7f50640a96b0 2024-03-25T19:19:49.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.115+0000 7f5060ff9640 1 -- 172.21.15.103:0/1744190091 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3953197026 0 0) 0x7f5044004eb0 con 0x7f5064142a20 2024-03-25T19:19:49.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.115+0000 7f5060ff9640 1 -- 172.21.15.103:0/1744190091 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5064133a00 con 0x7f5064142a20 2024-03-25T19:19:49.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.115+0000 7f5060ff9640 1 -- 172.21.15.103:0/1744190091 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f505c001f30 con 0x7f5064142a20 2024-03-25T19:19:49.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.115+0000 7f5060ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:49.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.115+0000 7f5060ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:19:49.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.115+0000 7f5060ff9640 10 monclient(hunting): dump: 2024-03-25T19:19:49.116 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 1 2024-03-25T19:19:49.116 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:49.116 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:49.116 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:49.116 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:49.116 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:49.117 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:49.117 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:49.117 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:49.117 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:49.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.115+0000 7f5060ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:19:49.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.115+0000 7f5060ff9640 1 -- 172.21.15.103:0/1744190091 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 624858817 0 0) 0x7f50640ac0e0 con 0x7f50640b4700 2024-03-25T19:19:49.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.116+0000 7f5060ff9640 1 -- 172.21.15.103:0/1744190091 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5044004eb0 con 0x7f50640b4700 2024-03-25T19:19:49.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.116+0000 7f5060ff9640 1 -- 172.21.15.103:0/1744190091 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f504c001870 con 0x7f50640b4700 2024-03-25T19:19:49.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.116+0000 7f5060ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:49.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.116+0000 7f5060ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:19:49.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.116+0000 7f5060ff9640 10 monclient(hunting): dump: 2024-03-25T19:19:49.117 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 1 2024-03-25T19:19:49.117 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:49.117 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:49.117 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:49.117 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:49.118 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:49.118 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:49.118 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:49.118 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:49.118 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:49.118 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.116+0000 7f5060ff9640 1 -- 172.21.15.103:0/1744190091 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3518353708 0 0) 0x7f506413c1f0 con 0x7f50640a96b0 2024-03-25T19:19:49.118 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.116+0000 7f5060ff9640 1 -- 172.21.15.103:0/1744190091 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50640ac0e0 con 0x7f50640a96b0 2024-03-25T19:19:49.118 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.116+0000 7f5060ff9640 1 -- 172.21.15.103:0/1744190091 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f50540018b0 con 0x7f50640a96b0 2024-03-25T19:19:49.118 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.116+0000 7f5060ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:49.118 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.116+0000 7f5060ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:19:49.118 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.116+0000 7f5060ff9640 10 monclient(hunting): dump: 2024-03-25T19:19:49.118 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 1 2024-03-25T19:19:49.118 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:49.118 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:49.118 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:49.118 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:49.118 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:49.119 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:49.119 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:49.119 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:49.119 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:49.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.119+0000 7f5060ff9640 1 -- 172.21.15.103:0/1744190091 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 667019023 0 0) 0x7f5064133a00 con 0x7f5064142a20 2024-03-25T19:19:49.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.119+0000 7f5060ff9640 1 -- 172.21.15.103:0/1744190091 >> v1:172.21.15.103:6789/0 conn(0x7f50640a96b0 legacy=0x7f5064138960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:49.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.119+0000 7f5060ff9640 1 -- 172.21.15.103:0/1744190091 >> v1:172.21.15.83:6789/0 conn(0x7f50640b4700 legacy=0x7f5064132be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:49.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.119+0000 7f5060ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:19:49.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.119+0000 7f5060ff9640 1 monclient: found mon.c 2024-03-25T19:19:49.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.119+0000 7f5060ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:19:49.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.119+0000 7f5060ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:19:49.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.119+0000 7f5060ff9640 1 -- 172.21.15.103:0/1744190091 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50640ac3c0 con 0x7f5064142a20 2024-03-25T19:19:49.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.119+0000 7f5060ff9640 10 monclient: _finish_auth 0 2024-03-25T19:19:49.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.119+0000 7f5060ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:19:49.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.119+0000 7f5060ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:19:49.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.119+0000 7f5069f82640 5 monclient: authenticate success, global_id 4145 2024-03-25T19:19:49.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.119+0000 7f5069f82640 10 monclient: _renew_subs 2024-03-25T19:19:49.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.119+0000 7f5069f82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:19:49.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.119+0000 7f5069f82640 1 -- 172.21.15.103:0/1744190091 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5064139070 con 0x7f5064142a20 2024-03-25T19:19:49.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.119+0000 7f5069f82640 10 client.4145.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:19:49.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.119+0000 7f5069f82640 10 monclient: _renew_subs 2024-03-25T19:19:49.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.119+0000 7f5069f82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:19:49.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.119+0000 7f5069f82640 1 -- 172.21.15.103:0/1744190091 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f506413c860 con 0x7f5064142a20 2024-03-25T19:19:49.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.119+0000 7f5069f82640 1 librados: init done 2024-03-25T19:19:49.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.120+0000 7f5069f82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:49.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.120+0000 7f5069f82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:49.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.120+0000 7f5069f82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:19:49.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.120+0000 7f5069f82640 1 -- 172.21.15.103:0/1744190091 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50640b3670 con 0x7f5064142a20 2024-03-25T19:19:49.143 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-03-25T19:19:49.144+0000 7f02a242b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T19:19:49.186 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-25T19:19:49.186+0000 7f54aa63c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:19:49.214 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.214+0000 7f5060ff9640 1 -- 172.21.15.103:0/1744190091 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f505c001590 con 0x7f5064142a20 2024-03-25T19:19:49.214 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.214+0000 7f5060ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:19:49.214 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.214+0000 7f5060ff9640 1 -- 172.21.15.103:0/1744190091 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f505c004fb0 con 0x7f5064142a20 2024-03-25T19:19:49.214 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.214+0000 7f5060ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:49.214 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.214+0000 7f5060ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:19:49.214 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.214+0000 7f5060ff9640 10 monclient: dump: 2024-03-25T19:19:49.214 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 1 2024-03-25T19:19:49.214 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:49.214 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:49.214 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:49.214 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:49.214 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:49.214 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:49.215 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:49.215 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:49.215 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:49.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.214+0000 7f5060ff9640 1 -- 172.21.15.103:0/1744190091 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f505c005520 con 0x7f5064142a20 2024-03-25T19:19:49.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.214+0000 7f50627fc640 4 set_mon_vals no callback set 2024-03-25T19:19:49.216 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.215+0000 7f5060ff9640 1 -- 172.21.15.103:0/1744190091 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 3581663666 0 0) 0x7f505c006b90 con 0x7f5064142a20 2024-03-25T19:19:49.216 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.215+0000 7f5060ff9640 10 client.4145.objecter ms_dispatch 0x7f5064000b90 osd_map(6..6 src has 1..6) v4 2024-03-25T19:19:49.216 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.215+0000 7f5060ff9640 3 client.4145.objecter handle_osd_map got epochs [6,6] > 0 2024-03-25T19:19:49.216 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.215+0000 7f5060ff9640 3 client.4145.objecter handle_osd_map decoding full epoch 6 2024-03-25T19:19:49.216 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.215+0000 7f5060ff9640 20 client.4145.objecter dump_active .. 0 homeless 2024-03-25T19:19:49.216 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.215+0000 7f5060ff9640 1 -- 172.21.15.103:0/1744190091 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f505c006680 con 0x7f5064142a20 2024-03-25T19:19:49.216 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.215+0000 7f5060ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:49.216 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.215+0000 7f5060ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:19:49.279 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-25T19:19:49.280+0000 7f54aa63c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T19:19:49.287 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.288+0000 7f5069f82640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "8354d57d-5c88-4259-995b-94c71f146960", "id": 4}] 2024-03-25T19:19:49.287 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.288+0000 7f5069f82640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "8354d57d-5c88-4259-995b-94c71f146960", "id": 4}] 2024-03-25T19:19:49.287 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.288+0000 7f5069f82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:19:49.287 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.288+0000 7f5069f82640 1 -- 172.21.15.103:0/1744190091 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd new", "uuid": "8354d57d-5c88-4259-995b-94c71f146960", "id": 4} v 0) v1 -- 0x7f506404f730 con 0x7f5064142a20 2024-03-25T19:19:49.301 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-03-25T19:19:49.302+0000 7f02a242b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T19:19:49.346 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.347+0000 7f5060ff9640 1 -- 172.21.15.103:0/1744190091 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8354d57d-5c88-4259-995b-94c71f146960", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 1051946684 0 1697012583) 0x7f505c0068a0 con 0x7f5064142a20 2024-03-25T19:19:49.347 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.347+0000 7f5060ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "8354d57d-5c88-4259-995b-94c71f146960", "id": 4}] 2024-03-25T19:19:49.347 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.347+0000 7f5060ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:19:49.347 INFO:teuthology.orchestra.run.smithi103.stdout:4 2024-03-25T19:19:49.348 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.348+0000 7f5069f82640 10 librados: watch_flush enter 2024-03-25T19:19:49.348 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.348+0000 7f5069f82640 10 librados: watch_flush exit 2024-03-25T19:19:49.348 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.348+0000 7f5069f82640 20 client.4145.objecter shutdown clearing up homeless session... 2024-03-25T19:19:49.348 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.348+0000 7f5069f82640 10 client.4145.objecter successfully canceled tick 2024-03-25T19:19:49.348 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.348+0000 7f5069f82640 10 monclient: shutdown 2024-03-25T19:19:49.348 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.348+0000 7f5069f82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:19:49.348 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.348+0000 7f5069f82640 1 -- 172.21.15.103:0/1744190091 >> v1:172.21.15.83:6790/0 conn(0x7f5064142a20 legacy=0x7f50641332f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:49.349 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.348+0000 7f5069f82640 1 -- 172.21.15.103:0/1744190091 shutdown_connections 2024-03-25T19:19:49.349 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.348+0000 7f5069f82640 1 -- 172.21.15.103:0/1744190091 >> 172.21.15.103:0/1744190091 conn(0x7f50640668e0 msgr2=0x7f50640b6dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:19:49.349 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.348+0000 7f5069f82640 1 -- 172.21.15.103:0/1744190091 shutdown_connections 2024-03-25T19:19:49.349 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.348+0000 7f5069f82640 1 -- 172.21.15.103:0/1744190091 wait complete. 2024-03-25T19:19:49.349 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.348+0000 7f5069f82640 1 librados: shutdown 2024-03-25T19:19:49.358 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new 801edd90-2540-4d68-aefd-f9766dff347c 5 2024-03-25T19:19:49.441 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-25T19:19:49.442+0000 7f54aa63c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:19:49.458 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.459+0000 7f950c9b5640 10 monclient: get_monmap_and_config 2024-03-25T19:19:49.458 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.459+0000 7f950c9b5640 10 monclient: build_initial_monmap 2024-03-25T19:19:49.458 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.459+0000 7f950c9b5640 10 monclient: monmap: 2024-03-25T19:19:49.458 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 0 2024-03-25T19:19:49.458 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:49.458 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:19:49.459696+0000 2024-03-25T19:19:49.459 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:19:49.459696+0000 2024-03-25T19:19:49.459 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 0 (unknown) 2024-03-25T19:19:49.459 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:49.459 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:19:49.459 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:19:49.459 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:19:49.459 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:49.460 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.459+0000 7f950c9b5640 1 Processor -- start 2024-03-25T19:19:49.460 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.459+0000 7f950c9b5640 1 -- start start 2024-03-25T19:19:49.460 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.459+0000 7f950c9b5640 10 monclient: init 2024-03-25T19:19:49.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.460+0000 7f950c9b5640 10 monclient: _reopen_session rank -1 2024-03-25T19:19:49.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.460+0000 7f950c9b5640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:19:49.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.460+0000 7f950c9b5640 10 monclient(hunting): picked mon.noname-a con 0x7f95080b4700 addr v1:172.21.15.83:6789/0 2024-03-25T19:19:49.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.460+0000 7f950c9b5640 10 monclient(hunting): picked mon.noname-c con 0x7f9508142a20 addr v1:172.21.15.103:6789/0 2024-03-25T19:19:49.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.460+0000 7f950c9b5640 10 monclient(hunting): picked mon.noname-b con 0x7f95080a96b0 addr v1:172.21.15.83:6790/0 2024-03-25T19:19:49.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.460+0000 7f950c9b5640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95080a9d60 con 0x7f95080b4700 2024-03-25T19:19:49.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.460+0000 7f950c9b5640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f950813ff10 con 0x7f9508142a20 2024-03-25T19:19:49.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.460+0000 7f950c9b5640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95080aa1f0 con 0x7f95080a96b0 2024-03-25T19:19:49.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.460+0000 7f950c9b5640 10 monclient(hunting): _renew_subs 2024-03-25T19:19:49.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.460+0000 7f950c9b5640 10 monclient(hunting): authenticate will time out at 6026.595211s 2024-03-25T19:19:49.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.461+0000 7f9506d9d640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f95080a96b0 0x7f950813f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.103:40496/0 (socket says 172.21.15.103:40496) 2024-03-25T19:19:49.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.461+0000 7f950659c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f95080b4700 0x7f9508142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.103:42366/0 (socket says 172.21.15.103:42366) 2024-03-25T19:19:49.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.461+0000 7f9506d9d640 1 -- 172.21.15.103:0/4244104310 learned_addr learned my addr 172.21.15.103:0/4244104310 (peer_addr_for_me v1:172.21.15.103:0/0) 2024-03-25T19:19:49.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.463+0000 7f9505d9b640 1 -- 172.21.15.103:0/4244104310 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3605893639 0 0) 0x7f95080a9d60 con 0x7f95080b4700 2024-03-25T19:19:49.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.463+0000 7f9505d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:19:49.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.463+0000 7f9505d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:49.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.463+0000 7f9505d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:49.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.463+0000 7f9505d9b640 10 monclient(hunting): my global_id is 4146 2024-03-25T19:19:49.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.463+0000 7f9505d9b640 1 -- 172.21.15.103:0/4244104310 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94f4003520 con 0x7f95080b4700 2024-03-25T19:19:49.464 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.463+0000 7f9505d9b640 1 -- 172.21.15.103:0/4244104310 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 151216064 0 0) 0x7f95080aa1f0 con 0x7f95080a96b0 2024-03-25T19:19:49.464 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.463+0000 7f9505d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:19:49.464 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.463+0000 7f9505d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:49.464 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.463+0000 7f9505d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:49.464 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.463+0000 7f9505d9b640 10 monclient(hunting): my global_id is 4148 2024-03-25T19:19:49.464 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.463+0000 7f9505d9b640 1 -- 172.21.15.103:0/4244104310 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95080a9d60 con 0x7f95080a96b0 2024-03-25T19:19:49.464 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.464+0000 7f9505d9b640 1 -- 172.21.15.103:0/4244104310 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3071771764 0 0) 0x7f950813ff10 con 0x7f9508142a20 2024-03-25T19:19:49.464 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.464+0000 7f9505d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:19:49.464 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.464+0000 7f9505d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:49.464 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.464+0000 7f9505d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:49.464 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.464+0000 7f9505d9b640 10 monclient(hunting): my global_id is 4147 2024-03-25T19:19:49.464 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.464+0000 7f9505d9b640 1 -- 172.21.15.103:0/4244104310 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95080aa1f0 con 0x7f9508142a20 2024-03-25T19:19:49.466 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.467+0000 7f9505d9b640 1 -- 172.21.15.103:0/4244104310 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 187092033 0 0) 0x7f94f4003520 con 0x7f95080b4700 2024-03-25T19:19:49.466 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.467+0000 7f9505d9b640 1 -- 172.21.15.103:0/4244104310 >> v1:172.21.15.83:6790/0 conn(0x7f95080a96b0 legacy=0x7f950813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:49.466 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.467+0000 7f9505d9b640 1 -- 172.21.15.103:0/4244104310 >> v1:172.21.15.103:6789/0 conn(0x7f9508142a20 legacy=0x7f950813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:49.467 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.467+0000 7f9505d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:19:49.467 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.467+0000 7f9505d9b640 1 monclient: found mon.noname-a 2024-03-25T19:19:49.467 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.467+0000 7f9505d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:19:49.467 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.467+0000 7f9505d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.83:6789/0 2024-03-25T19:19:49.467 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.467+0000 7f9505d9b640 1 -- 172.21.15.103:0/4244104310 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95080b4ae0 con 0x7f95080b4700 2024-03-25T19:19:49.467 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.467+0000 7f9505d9b640 10 monclient: _finish_auth 0 2024-03-25T19:19:49.467 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.467+0000 7f9505d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:19:49.467 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.467+0000 7f9505d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:19:49.467 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.467+0000 7f9505d9b640 1 -- 172.21.15.103:0/4244104310 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f94f8001690 con 0x7f95080b4700 2024-03-25T19:19:49.467 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.467+0000 7f9505d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:49.467 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.467+0000 7f9505d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:19:49.467 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.467+0000 7f9505d9b640 10 monclient: dump: 2024-03-25T19:19:49.467 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 1 2024-03-25T19:19:49.467 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:49.467 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:49.468 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:49.468 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:49.468 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:49.468 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:49.468 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:49.468 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:49.468 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:49.468 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.467+0000 7f950c9b5640 5 monclient: authenticate success, global_id 4146 2024-03-25T19:19:49.468 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.467+0000 7f950c9b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:19:49.471 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.468+0000 7f9505d9b640 1 -- 172.21.15.103:0/4244104310 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f94f80017f0 con 0x7f95080b4700 2024-03-25T19:19:49.471 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.468+0000 7f9505d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:19:49.471 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.468+0000 7f950c9b5640 10 monclient: get_monmap_and_config success 2024-03-25T19:19:49.471 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.468+0000 7f9505d9b640 1 -- 172.21.15.103:0/4244104310 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f94f8001c00 con 0x7f95080b4700 2024-03-25T19:19:49.471 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.468+0000 7f950c9b5640 4 set_mon_vals no callback set 2024-03-25T19:19:49.471 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.468+0000 7f9505d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:49.471 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.468+0000 7f9505d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:19:49.472 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.468+0000 7f9505d9b640 10 monclient: dump: 2024-03-25T19:19:49.472 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 1 2024-03-25T19:19:49.472 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:49.472 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:49.472 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:49.472 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:49.472 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:49.472 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:49.472 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:49.472 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:49.472 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:49.472 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.469+0000 7f950c9b5640 10 monclient: shutdown 2024-03-25T19:19:49.472 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.469+0000 7f950c9b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:19:49.472 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.469+0000 7f950c9b5640 1 -- 172.21.15.103:0/4244104310 >> v1:172.21.15.83:6789/0 conn(0x7f95080b4700 legacy=0x7f9508142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:49.472 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.469+0000 7f950c9b5640 1 -- 172.21.15.103:0/4244104310 shutdown_connections 2024-03-25T19:19:49.473 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.469+0000 7f950c9b5640 1 -- 172.21.15.103:0/4244104310 >> 172.21.15.103:0/4244104310 conn(0x7f95080668e0 msgr2=0x7f9508066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:19:49.473 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.469+0000 7f950c9b5640 1 -- 172.21.15.103:0/4244104310 shutdown_connections 2024-03-25T19:19:49.473 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.469+0000 7f950c9b5640 1 -- 172.21.15.103:0/4244104310 wait complete. 2024-03-25T19:19:49.473 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.470+0000 7f950c9b5640 10 monclient: build_initial_monmap 2024-03-25T19:19:49.473 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.470+0000 7f950c9b5640 10 monclient: monmap: 2024-03-25T19:19:49.473 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 0 2024-03-25T19:19:49.473 INFO:teuthology.orchestra.run.smithi103.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:19:49.473 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 0.000000 2024-03-25T19:19:49.473 INFO:teuthology.orchestra.run.smithi103.stderr:created 0.000000 2024-03-25T19:19:49.473 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 0 (unknown) 2024-03-25T19:19:49.473 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 1 2024-03-25T19:19:49.473 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:19:49.473 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:19:49.473 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:19:49.473 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:49.474 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.470+0000 7f950c9b5640 1 librados: starting msgr at 2024-03-25T19:19:49.474 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.470+0000 7f950c9b5640 1 librados: starting objecter 2024-03-25T19:19:49.474 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.470+0000 7f950c9b5640 1 Processor -- start 2024-03-25T19:19:49.474 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.470+0000 7f950c9b5640 1 -- start start 2024-03-25T19:19:49.474 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.470+0000 7f950c9b5640 1 librados: setting wanted keys 2024-03-25T19:19:49.474 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.470+0000 7f950c9b5640 1 librados: calling monclient init 2024-03-25T19:19:49.474 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.470+0000 7f950c9b5640 10 monclient: init 2024-03-25T19:19:49.474 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.471+0000 7f950c9b5640 10 monclient: _reopen_session rank -1 2024-03-25T19:19:49.474 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.471+0000 7f950c9b5640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:19:49.474 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.471+0000 7f950c9b5640 10 monclient(hunting): picked mon.noname-a con 0x7f95080a96b0 addr v1:172.21.15.83:6789/0 2024-03-25T19:19:49.474 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.471+0000 7f950c9b5640 10 monclient(hunting): picked mon.noname-b con 0x7f95080b4700 addr v1:172.21.15.103:6789/0 2024-03-25T19:19:49.474 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.471+0000 7f950c9b5640 10 monclient(hunting): picked mon.noname-c con 0x7f9508142a20 addr v1:172.21.15.83:6790/0 2024-03-25T19:19:49.474 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.471+0000 7f950c9b5640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95080ac8c0 con 0x7f95080a96b0 2024-03-25T19:19:49.474 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.471+0000 7f950c9b5640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95080adfb0 con 0x7f95080b4700 2024-03-25T19:19:49.475 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.471+0000 7f950c9b5640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95080ae290 con 0x7f9508142a20 2024-03-25T19:19:49.475 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.471+0000 7f950c9b5640 10 monclient(hunting): _renew_subs 2024-03-25T19:19:49.475 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.471+0000 7f950c9b5640 10 monclient(hunting): authenticate will time out at 6026.606008s 2024-03-25T19:19:49.475 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.471+0000 7f950659c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f95080a96b0 0x7f95080abea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.103:42378/0 (socket says 172.21.15.103:42378) 2024-03-25T19:19:49.475 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.471+0000 7f950659c640 1 -- 172.21.15.103:0/1164014745 learned_addr learned my addr 172.21.15.103:0/1164014745 (peer_addr_for_me v1:172.21.15.103:0/0) 2024-03-25T19:19:49.475 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.472+0000 7f94feffd640 10 client.?.objecter ms_handle_connect 0x7f9508142a20 2024-03-25T19:19:49.475 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.472+0000 7f94feffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:49.475 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.472+0000 7f94feffd640 10 client.?.objecter ms_handle_connect 0x7f95080a96b0 2024-03-25T19:19:49.475 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.472+0000 7f94feffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:49.476 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.474+0000 7f94feffd640 1 -- 172.21.15.103:0/1164014745 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 7224055 0 0) 0x7f95080ac8c0 con 0x7f95080a96b0 2024-03-25T19:19:49.476 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.474+0000 7f94feffd640 10 monclient(hunting): _negotiate 2024-03-25T19:19:49.476 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.474+0000 7f94feffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:49.476 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.474+0000 7f94feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:49.476 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.474+0000 7f94feffd640 10 monclient(hunting): my global_id is 4149 2024-03-25T19:19:49.476 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.474+0000 7f94feffd640 1 -- 172.21.15.103:0/1164014745 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94e8004eb0 con 0x7f95080a96b0 2024-03-25T19:19:49.476 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.474+0000 7f94feffd640 1 -- 172.21.15.103:0/1164014745 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1639284527 0 0) 0x7f95080ae290 con 0x7f9508142a20 2024-03-25T19:19:49.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.474+0000 7f94feffd640 10 monclient(hunting): _negotiate 2024-03-25T19:19:49.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.474+0000 7f94feffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:49.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.474+0000 7f94feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:49.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.474+0000 7f94feffd640 10 monclient(hunting): my global_id is 4151 2024-03-25T19:19:49.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.474+0000 7f94feffd640 1 -- 172.21.15.103:0/1164014745 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95080ac8c0 con 0x7f9508142a20 2024-03-25T19:19:49.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.476+0000 7f94feffd640 10 client.?.objecter ms_handle_connect 0x7f95080b4700 2024-03-25T19:19:49.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.476+0000 7f94feffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:49.479 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.479+0000 7f94feffd640 1 -- 172.21.15.103:0/1164014745 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1766224950 0 0) 0x7f95080adfb0 con 0x7f95080b4700 2024-03-25T19:19:49.479 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.479+0000 7f94feffd640 10 monclient(hunting): _negotiate 2024-03-25T19:19:49.479 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.479+0000 7f94feffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:49.479 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.479+0000 7f94feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:49.479 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.479+0000 7f94feffd640 10 monclient(hunting): my global_id is 4150 2024-03-25T19:19:49.479 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.479+0000 7f94feffd640 1 -- 172.21.15.103:0/1164014745 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95080ae290 con 0x7f95080b4700 2024-03-25T19:19:49.479 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.479+0000 7f94feffd640 1 -- 172.21.15.103:0/1164014745 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1704788832 0 0) 0x7f94e8004eb0 con 0x7f95080a96b0 2024-03-25T19:19:49.479 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.479+0000 7f94feffd640 1 -- 172.21.15.103:0/1164014745 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95080adfb0 con 0x7f95080a96b0 2024-03-25T19:19:49.479 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.479+0000 7f94feffd640 1 -- 172.21.15.103:0/1164014745 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f94f8001bc0 con 0x7f95080a96b0 2024-03-25T19:19:49.479 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.479+0000 7f94feffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:49.480 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.479+0000 7f94feffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:19:49.480 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.479+0000 7f94feffd640 10 monclient(hunting): dump: 2024-03-25T19:19:49.480 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 1 2024-03-25T19:19:49.480 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:49.480 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:49.480 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:49.480 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:49.480 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:49.480 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:49.480 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:49.480 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:49.480 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:49.480 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.479+0000 7f94feffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:19:49.480 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.479+0000 7f94feffd640 1 -- 172.21.15.103:0/1164014745 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2378362317 0 0) 0x7f95080ac8c0 con 0x7f9508142a20 2024-03-25T19:19:49.480 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.479+0000 7f94feffd640 1 -- 172.21.15.103:0/1164014745 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94e8004eb0 con 0x7f9508142a20 2024-03-25T19:19:49.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.479+0000 7f94feffd640 1 -- 172.21.15.103:0/1164014745 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f9500001bd0 con 0x7f9508142a20 2024-03-25T19:19:49.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.479+0000 7f94feffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:49.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.479+0000 7f94feffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:19:49.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.479+0000 7f94feffd640 10 monclient(hunting): dump: 2024-03-25T19:19:49.481 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 1 2024-03-25T19:19:49.481 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:49.481 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:49.481 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:49.481 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:49.481 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:49.481 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:49.481 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:49.481 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:49.481 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:49.483 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.482+0000 7f94feffd640 1 -- 172.21.15.103:0/1164014745 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 11263304 0 0) 0x7f94e8004eb0 con 0x7f9508142a20 2024-03-25T19:19:49.483 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.482+0000 7f94feffd640 1 -- 172.21.15.103:0/1164014745 >> v1:172.21.15.103:6789/0 conn(0x7f95080b4700 legacy=0x7f95080ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:49.483 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.482+0000 7f94feffd640 1 -- 172.21.15.103:0/1164014745 >> v1:172.21.15.83:6789/0 conn(0x7f95080a96b0 legacy=0x7f95080abea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:49.483 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.482+0000 7f94feffd640 10 monclient: _finish_hunting 0 2024-03-25T19:19:49.483 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.482+0000 7f94feffd640 1 monclient: found mon.c 2024-03-25T19:19:49.483 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.482+0000 7f94feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:19:49.483 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.482+0000 7f94feffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:19:49.483 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.482+0000 7f94feffd640 1 -- 172.21.15.103:0/1164014745 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95080aedc0 con 0x7f9508142a20 2024-03-25T19:19:49.483 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.482+0000 7f94feffd640 10 monclient: _finish_auth 0 2024-03-25T19:19:49.483 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.482+0000 7f94feffd640 10 monclient: _check_auth_tickets 2024-03-25T19:19:49.483 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.482+0000 7f94feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:19:49.483 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.482+0000 7f950c9b5640 5 monclient: authenticate success, global_id 4151 2024-03-25T19:19:49.483 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.482+0000 7f950c9b5640 10 monclient: _renew_subs 2024-03-25T19:19:49.484 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.482+0000 7f950c9b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:19:49.484 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.482+0000 7f950c9b5640 1 -- 172.21.15.103:0/1164014745 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f950813ae40 con 0x7f9508142a20 2024-03-25T19:19:49.484 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.482+0000 7f950c9b5640 10 client.4151.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:19:49.484 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.482+0000 7f950c9b5640 10 monclient: _renew_subs 2024-03-25T19:19:49.484 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.482+0000 7f950c9b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:19:49.484 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.482+0000 7f950c9b5640 1 -- 172.21.15.103:0/1164014745 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f95080acf30 con 0x7f9508142a20 2024-03-25T19:19:49.484 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.482+0000 7f950c9b5640 1 librados: init done 2024-03-25T19:19:49.484 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.483+0000 7f950c9b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:49.484 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.483+0000 7f950c9b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:49.484 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.483+0000 7f950c9b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:19:49.484 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.483+0000 7f950c9b5640 1 -- 172.21.15.103:0/1164014745 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9508135ea0 con 0x7f9508142a20 2024-03-25T19:19:49.485 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-25T19:19:49.486+0000 7f54aa63c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T19:19:49.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.580+0000 7f94feffd640 1 -- 172.21.15.103:0/1164014745 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9500002080 con 0x7f9508142a20 2024-03-25T19:19:49.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.580+0000 7f94feffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:19:49.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.580+0000 7f94feffd640 1 -- 172.21.15.103:0/1164014745 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f9500004b60 con 0x7f9508142a20 2024-03-25T19:19:49.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.580+0000 7f94feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:49.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.580+0000 7f94feffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:19:49.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.580+0000 7f94feffd640 10 monclient: dump: 2024-03-25T19:19:49.579 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 1 2024-03-25T19:19:49.579 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:49.580 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:49.580 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:49.580 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:49.580 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:49.580 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:49.580 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:49.580 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:49.580 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:49.580 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.580+0000 7f94feffd640 1 -- 172.21.15.103:0/1164014745 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f95000050d0 con 0x7f9508142a20 2024-03-25T19:19:49.580 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.580+0000 7f950559a640 4 set_mon_vals no callback set 2024-03-25T19:19:49.581 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.582+0000 7f94feffd640 1 -- 172.21.15.103:0/1164014745 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 2245843248 0 0) 0x7f9500006870 con 0x7f9508142a20 2024-03-25T19:19:49.581 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.582+0000 7f94feffd640 10 client.4151.objecter ms_dispatch 0x7f9508000b90 osd_map(7..7 src has 1..7) v4 2024-03-25T19:19:49.581 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.582+0000 7f94feffd640 3 client.4151.objecter handle_osd_map got epochs [7,7] > 0 2024-03-25T19:19:49.581 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.582+0000 7f94feffd640 3 client.4151.objecter handle_osd_map decoding full epoch 7 2024-03-25T19:19:49.581 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.582+0000 7f94feffd640 20 client.4151.objecter dump_active .. 0 homeless 2024-03-25T19:19:49.581 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.582+0000 7f94feffd640 1 -- 172.21.15.103:0/1164014745 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f9500006b40 con 0x7f9508142a20 2024-03-25T19:19:49.582 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.582+0000 7f94feffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:49.582 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.582+0000 7f94feffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:19:49.642 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-25T19:19:49.643+0000 7f54aa63c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T19:19:49.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.655+0000 7f950c9b5640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "801edd90-2540-4d68-aefd-f9766dff347c", "id": 5}] 2024-03-25T19:19:49.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.655+0000 7f950c9b5640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "801edd90-2540-4d68-aefd-f9766dff347c", "id": 5}] 2024-03-25T19:19:49.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.655+0000 7f950c9b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:19:49.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.655+0000 7f950c9b5640 1 -- 172.21.15.103:0/1164014745 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd new", "uuid": "801edd90-2540-4d68-aefd-f9766dff347c", "id": 5} v 0) v1 -- 0x7f950804f730 con 0x7f9508142a20 2024-03-25T19:19:49.726 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.726+0000 7f94feffd640 1 -- 172.21.15.103:0/1164014745 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "801edd90-2540-4d68-aefd-f9766dff347c", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 3507975997 0 1988412176) 0x7f95000062f0 con 0x7f9508142a20 2024-03-25T19:19:49.726 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.726+0000 7f94feffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "801edd90-2540-4d68-aefd-f9766dff347c", "id": 5}] 2024-03-25T19:19:49.726 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.726+0000 7f94feffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:19:49.726 INFO:teuthology.orchestra.run.smithi103.stdout:5 2024-03-25T19:19:49.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.727+0000 7f950c9b5640 10 librados: watch_flush enter 2024-03-25T19:19:49.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.727+0000 7f950c9b5640 10 librados: watch_flush exit 2024-03-25T19:19:49.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.727+0000 7f950c9b5640 20 client.4151.objecter shutdown clearing up homeless session... 2024-03-25T19:19:49.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.727+0000 7f950c9b5640 10 client.4151.objecter successfully canceled tick 2024-03-25T19:19:49.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.727+0000 7f950c9b5640 10 monclient: shutdown 2024-03-25T19:19:49.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.727+0000 7f950c9b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:19:49.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.727+0000 7f950c9b5640 1 -- 172.21.15.103:0/1164014745 >> v1:172.21.15.83:6790/0 conn(0x7f9508142a20 legacy=0x7f95081e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:49.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.727+0000 7f950c9b5640 1 -- 172.21.15.103:0/1164014745 shutdown_connections 2024-03-25T19:19:49.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.727+0000 7f950c9b5640 1 -- 172.21.15.103:0/1164014745 >> 172.21.15.103:0/1164014745 conn(0x7f95080668e0 msgr2=0x7f95080b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:19:49.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.727+0000 7f950c9b5640 1 -- 172.21.15.103:0/1164014745 shutdown_connections 2024-03-25T19:19:49.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.727+0000 7f950c9b5640 1 -- 172.21.15.103:0/1164014745 wait complete. 2024-03-25T19:19:49.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.727+0000 7f950c9b5640 1 librados: shutdown 2024-03-25T19:19:49.737 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new 56c6dbb7-33a7-4af1-a6ab-26c25e08f14c 6 2024-03-25T19:19:49.837 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.837+0000 7f5c758f6640 10 monclient: get_monmap_and_config 2024-03-25T19:19:49.837 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.837+0000 7f5c758f6640 10 monclient: build_initial_monmap 2024-03-25T19:19:49.837 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.837+0000 7f5c758f6640 10 monclient: monmap: 2024-03-25T19:19:49.837 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 0 2024-03-25T19:19:49.837 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:49.837 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:19:49.838300+0000 2024-03-25T19:19:49.837 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:19:49.838300+0000 2024-03-25T19:19:49.837 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 0 (unknown) 2024-03-25T19:19:49.837 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:49.837 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:19:49.837 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:19:49.837 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:19:49.837 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:49.839 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.838+0000 7f5c758f6640 1 Processor -- start 2024-03-25T19:19:49.839 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.838+0000 7f5c758f6640 1 -- start start 2024-03-25T19:19:49.839 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.838+0000 7f5c758f6640 10 monclient: init 2024-03-25T19:19:49.839 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.838+0000 7f5c758f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:19:49.839 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.838+0000 7f5c758f6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:19:49.839 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.838+0000 7f5c758f6640 10 monclient(hunting): picked mon.noname-a con 0x7f5c7006c2d0 addr v1:172.21.15.83:6789/0 2024-03-25T19:19:49.839 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.838+0000 7f5c758f6640 10 monclient(hunting): picked mon.noname-c con 0x7f5c7006c6b0 addr v1:172.21.15.103:6789/0 2024-03-25T19:19:49.839 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.838+0000 7f5c758f6640 10 monclient(hunting): picked mon.noname-b con 0x7f5c70142d90 addr v1:172.21.15.83:6790/0 2024-03-25T19:19:49.839 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.838+0000 7f5c758f6640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c700b0840 con 0x7f5c7006c2d0 2024-03-25T19:19:49.839 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.838+0000 7f5c758f6640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c70143200 con 0x7f5c7006c6b0 2024-03-25T19:19:49.840 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.838+0000 7f5c758f6640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c7006d810 con 0x7f5c70142d90 2024-03-25T19:19:49.840 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.838+0000 7f5c758f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:19:49.840 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.838+0000 7f5c758f6640 10 monclient(hunting): authenticate will time out at 6026.973569s 2024-03-25T19:19:49.840 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.839+0000 7f5c6effd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f5c7006c2d0 0x7f5c700afde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.103:42394/0 (socket says 172.21.15.103:42394) 2024-03-25T19:19:49.840 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.839+0000 7f5c6effd640 1 -- 172.21.15.103:0/1460256422 learned_addr learned my addr 172.21.15.103:0/1460256422 (peer_addr_for_me v1:172.21.15.103:0/0) 2024-03-25T19:19:49.840 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.839+0000 7f5c6f7fe640 1 --1- 172.21.15.103:0/1460256422 >> v1:172.21.15.83:6790/0 conn(0x7f5c70142d90 0x7f5c7014dbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.103:40514/0 (socket says 172.21.15.103:40514) 2024-03-25T19:19:49.840 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.841+0000 7f5c6dffb640 1 -- 172.21.15.103:0/1460256422 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1696043011 0 0) 0x7f5c64001040 con 0x7f5c7006c6b0 2024-03-25T19:19:49.841 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.841+0000 7f5c6dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:19:49.841 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.841+0000 7f5c6dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:49.841 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.841+0000 7f5c6dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:49.841 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.841+0000 7f5c6dffb640 10 monclient(hunting): my global_id is 4153 2024-03-25T19:19:49.841 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.841+0000 7f5c6dffb640 1 -- 172.21.15.103:0/1460256422 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c58003520 con 0x7f5c7006c6b0 2024-03-25T19:19:49.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.842+0000 7f5c6dffb640 1 -- 172.21.15.103:0/1460256422 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4248427802 0 0) 0x7f5c700b0840 con 0x7f5c7006c2d0 2024-03-25T19:19:49.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.842+0000 7f5c6dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:19:49.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.842+0000 7f5c6dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:49.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.842+0000 7f5c6dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:49.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.842+0000 7f5c6dffb640 10 monclient(hunting): my global_id is 4152 2024-03-25T19:19:49.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.842+0000 7f5c6dffb640 1 -- 172.21.15.103:0/1460256422 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c64001040 con 0x7f5c7006c2d0 2024-03-25T19:19:49.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.842+0000 7f5c6dffb640 1 -- 172.21.15.103:0/1460256422 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1604998788 0 0) 0x7f5c7006d810 con 0x7f5c70142d90 2024-03-25T19:19:49.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.842+0000 7f5c6dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:19:49.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.842+0000 7f5c6dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:49.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.842+0000 7f5c6dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:49.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.842+0000 7f5c6dffb640 10 monclient(hunting): my global_id is 4154 2024-03-25T19:19:49.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.842+0000 7f5c6dffb640 1 -- 172.21.15.103:0/1460256422 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c700b0840 con 0x7f5c70142d90 2024-03-25T19:19:49.844 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.844+0000 7f5c6dffb640 1 -- 172.21.15.103:0/1460256422 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 262944487 0 0) 0x7f5c58003520 con 0x7f5c7006c6b0 2024-03-25T19:19:49.844 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.844+0000 7f5c6dffb640 1 -- 172.21.15.103:0/1460256422 >> v1:172.21.15.83:6790/0 conn(0x7f5c70142d90 legacy=0x7f5c7014dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:49.844 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.844+0000 7f5c6dffb640 1 -- 172.21.15.103:0/1460256422 >> v1:172.21.15.83:6789/0 conn(0x7f5c7006c2d0 legacy=0x7f5c700afde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:49.844 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.844+0000 7f5c6dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:19:49.844 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.844+0000 7f5c6dffb640 1 monclient: found mon.noname-c 2024-03-25T19:19:49.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.844+0000 7f5c6dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:19:49.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.844+0000 7f5c6dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.103:6789/0 2024-03-25T19:19:49.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.844+0000 7f5c6dffb640 1 -- 172.21.15.103:0/1460256422 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c700ae260 con 0x7f5c7006c6b0 2024-03-25T19:19:49.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.844+0000 7f5c6dffb640 10 monclient: _finish_auth 0 2024-03-25T19:19:49.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.844+0000 7f5c6dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:19:49.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.844+0000 7f5c6dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:19:49.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.844+0000 7f5c6dffb640 1 -- 172.21.15.103:0/1460256422 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f5c640015b0 con 0x7f5c7006c6b0 2024-03-25T19:19:49.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.844+0000 7f5c6dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:49.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.844+0000 7f5c6dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:19:49.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.844+0000 7f5c6dffb640 10 monclient: dump: 2024-03-25T19:19:49.845 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 1 2024-03-25T19:19:49.845 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:49.845 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:49.845 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:49.845 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:49.846 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:49.846 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:49.846 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:49.846 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:49.846 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:49.846 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.845+0000 7f5c758f6640 5 monclient: authenticate success, global_id 4153 2024-03-25T19:19:49.846 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.845+0000 7f5c758f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:19:49.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.846+0000 7f5c6dffb640 1 -- 172.21.15.103:0/1460256422 <== mon.1 v1:172.21.15.103:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c64001710 con 0x7f5c7006c6b0 2024-03-25T19:19:49.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.846+0000 7f5c6dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:19:49.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.846+0000 7f5c6dffb640 1 -- 172.21.15.103:0/1460256422 <== mon.1 v1:172.21.15.103:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f5c64001aa0 con 0x7f5c7006c6b0 2024-03-25T19:19:49.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.846+0000 7f5c6dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:49.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.846+0000 7f5c6dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:19:49.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.846+0000 7f5c6dffb640 10 monclient: dump: 2024-03-25T19:19:49.849 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 1 2024-03-25T19:19:49.849 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:49.849 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:49.849 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:49.849 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:49.849 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:49.849 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:49.849 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:49.849 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:49.849 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:49.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.846+0000 7f5c758f6640 10 monclient: get_monmap_and_config success 2024-03-25T19:19:49.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.846+0000 7f5c758f6640 4 set_mon_vals no callback set 2024-03-25T19:19:49.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.847+0000 7f5c758f6640 10 monclient: shutdown 2024-03-25T19:19:49.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.847+0000 7f5c758f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:19:49.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.847+0000 7f5c758f6640 1 -- 172.21.15.103:0/1460256422 >> v1:172.21.15.103:6789/0 conn(0x7f5c7006c6b0 legacy=0x7f5c70142680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:49.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.847+0000 7f5c758f6640 1 -- 172.21.15.103:0/1460256422 shutdown_connections 2024-03-25T19:19:49.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.847+0000 7f5c758f6640 1 -- 172.21.15.103:0/1460256422 >> 172.21.15.103:0/1460256422 conn(0x7f5c700b5e40 msgr2=0x7f5c700b6250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:19:49.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.847+0000 7f5c758f6640 1 -- 172.21.15.103:0/1460256422 shutdown_connections 2024-03-25T19:19:49.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.847+0000 7f5c758f6640 1 -- 172.21.15.103:0/1460256422 wait complete. 2024-03-25T19:19:49.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.847+0000 7f5c758f6640 10 monclient: build_initial_monmap 2024-03-25T19:19:49.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.847+0000 7f5c758f6640 10 monclient: monmap: 2024-03-25T19:19:49.850 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 0 2024-03-25T19:19:49.850 INFO:teuthology.orchestra.run.smithi103.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:19:49.850 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 0.000000 2024-03-25T19:19:49.850 INFO:teuthology.orchestra.run.smithi103.stderr:created 0.000000 2024-03-25T19:19:49.850 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 0 (unknown) 2024-03-25T19:19:49.850 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 1 2024-03-25T19:19:49.850 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:19:49.851 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:19:49.851 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:19:49.851 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:49.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.848+0000 7f5c758f6640 1 librados: starting msgr at 2024-03-25T19:19:49.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.848+0000 7f5c758f6640 1 librados: starting objecter 2024-03-25T19:19:49.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.848+0000 7f5c758f6640 1 Processor -- start 2024-03-25T19:19:49.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.848+0000 7f5c758f6640 1 -- start start 2024-03-25T19:19:49.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.848+0000 7f5c758f6640 1 librados: setting wanted keys 2024-03-25T19:19:49.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.848+0000 7f5c758f6640 1 librados: calling monclient init 2024-03-25T19:19:49.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.848+0000 7f5c758f6640 10 monclient: init 2024-03-25T19:19:49.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.849+0000 7f5c758f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:19:49.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.849+0000 7f5c758f6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:19:49.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.849+0000 7f5c758f6640 10 monclient(hunting): picked mon.noname-b con 0x7f5c7006c2d0 addr v1:172.21.15.103:6789/0 2024-03-25T19:19:49.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.849+0000 7f5c758f6640 10 monclient(hunting): picked mon.noname-c con 0x7f5c7006c6b0 addr v1:172.21.15.83:6790/0 2024-03-25T19:19:49.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.849+0000 7f5c758f6640 10 monclient(hunting): picked mon.noname-a con 0x7f5c70142d90 addr v1:172.21.15.83:6789/0 2024-03-25T19:19:49.852 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.849+0000 7f5c758f6640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c700714d0 con 0x7f5c70142d90 2024-03-25T19:19:49.852 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.849+0000 7f5c758f6640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c70141aa0 con 0x7f5c7006c2d0 2024-03-25T19:19:49.852 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.849+0000 7f5c758f6640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c7013f9f0 con 0x7f5c7006c6b0 2024-03-25T19:19:49.852 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.849+0000 7f5c758f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:19:49.852 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.849+0000 7f5c758f6640 10 monclient(hunting): authenticate will time out at 6026.983807s 2024-03-25T19:19:49.852 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.849+0000 7f5c6f7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f5c70142d90 0x7f5c701e1e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.103:42410/0 (socket says 172.21.15.103:42410) 2024-03-25T19:19:49.852 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.849+0000 7f5c6f7fe640 1 -- 172.21.15.103:0/3648670232 learned_addr learned my addr 172.21.15.103:0/3648670232 (peer_addr_for_me v1:172.21.15.103:0/0) 2024-03-25T19:19:49.853 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.849+0000 7f5c57fff640 10 client.?.objecter ms_handle_connect 0x7f5c70142d90 2024-03-25T19:19:49.853 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.849+0000 7f5c57fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:49.853 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.849+0000 7f5c57fff640 10 client.?.objecter ms_handle_connect 0x7f5c7006c6b0 2024-03-25T19:19:49.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.849+0000 7f5c57fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:49.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.850+0000 7f5c57fff640 10 client.?.objecter ms_handle_connect 0x7f5c7006c2d0 2024-03-25T19:19:49.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.850+0000 7f5c57fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:49.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.852+0000 7f5c57fff640 1 -- 172.21.15.103:0/3648670232 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4048234333 0 0) 0x7f5c70141aa0 con 0x7f5c7006c2d0 2024-03-25T19:19:49.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.852+0000 7f5c57fff640 10 monclient(hunting): _negotiate 2024-03-25T19:19:49.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.852+0000 7f5c57fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:49.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.852+0000 7f5c57fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:49.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.852+0000 7f5c57fff640 10 monclient(hunting): my global_id is 4156 2024-03-25T19:19:49.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.852+0000 7f5c57fff640 1 -- 172.21.15.103:0/3648670232 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c5c004eb0 con 0x7f5c7006c2d0 2024-03-25T19:19:49.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.853+0000 7f5c57fff640 1 -- 172.21.15.103:0/3648670232 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 977634741 0 0) 0x7f5c7013f9f0 con 0x7f5c7006c6b0 2024-03-25T19:19:49.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.853+0000 7f5c57fff640 10 monclient(hunting): _negotiate 2024-03-25T19:19:49.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.853+0000 7f5c57fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:49.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.853+0000 7f5c57fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:49.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.853+0000 7f5c57fff640 10 monclient(hunting): my global_id is 4157 2024-03-25T19:19:49.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.853+0000 7f5c57fff640 1 -- 172.21.15.103:0/3648670232 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c70141aa0 con 0x7f5c7006c6b0 2024-03-25T19:19:49.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.854+0000 7f5c57fff640 1 -- 172.21.15.103:0/3648670232 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 707098578 0 0) 0x7f5c700714d0 con 0x7f5c70142d90 2024-03-25T19:19:49.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.854+0000 7f5c57fff640 10 monclient(hunting): _negotiate 2024-03-25T19:19:49.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.854+0000 7f5c57fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:49.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.854+0000 7f5c57fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:49.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.854+0000 7f5c57fff640 10 monclient(hunting): my global_id is 4155 2024-03-25T19:19:49.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.854+0000 7f5c57fff640 1 -- 172.21.15.103:0/3648670232 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c7013f9f0 con 0x7f5c70142d90 2024-03-25T19:19:49.856 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.857+0000 7f5c57fff640 1 -- 172.21.15.103:0/3648670232 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 984948562 0 0) 0x7f5c5c004eb0 con 0x7f5c7006c2d0 2024-03-25T19:19:49.857 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.857+0000 7f5c57fff640 1 -- 172.21.15.103:0/3648670232 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c700714d0 con 0x7f5c7006c2d0 2024-03-25T19:19:49.857 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.857+0000 7f5c57fff640 1 -- 172.21.15.103:0/3648670232 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f5c60001840 con 0x7f5c7006c2d0 2024-03-25T19:19:49.857 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.857+0000 7f5c57fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:49.857 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.857+0000 7f5c57fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:19:49.857 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.857+0000 7f5c57fff640 10 monclient(hunting): dump: 2024-03-25T19:19:49.857 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 1 2024-03-25T19:19:49.857 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:49.857 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:49.857 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:49.857 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:49.857 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:49.857 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:49.857 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:49.857 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:49.858 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:49.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.857+0000 7f5c57fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:19:49.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.858+0000 7f5c57fff640 1 -- 172.21.15.103:0/3648670232 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 214707504 0 0) 0x7f5c70141aa0 con 0x7f5c7006c6b0 2024-03-25T19:19:49.859 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.858+0000 7f5c57fff640 1 -- 172.21.15.103:0/3648670232 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c5c004eb0 con 0x7f5c7006c6b0 2024-03-25T19:19:49.859 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.858+0000 7f5c57fff640 1 -- 172.21.15.103:0/3648670232 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f5c64001d60 con 0x7f5c7006c6b0 2024-03-25T19:19:49.859 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.858+0000 7f5c57fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:49.859 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.858+0000 7f5c57fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:19:49.859 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.858+0000 7f5c57fff640 10 monclient(hunting): dump: 2024-03-25T19:19:49.859 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 1 2024-03-25T19:19:49.859 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:49.859 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:49.859 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:49.859 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:49.859 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:49.859 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:49.859 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:49.859 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:49.859 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:49.860 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.858+0000 7f5c57fff640 1 -- 172.21.15.103:0/3648670232 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1594271405 0 0) 0x7f5c7013f9f0 con 0x7f5c70142d90 2024-03-25T19:19:49.860 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.858+0000 7f5c57fff640 1 -- 172.21.15.103:0/3648670232 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c70141aa0 con 0x7f5c70142d90 2024-03-25T19:19:49.860 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.858+0000 7f5c57fff640 1 -- 172.21.15.103:0/3648670232 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f5c68001bf0 con 0x7f5c70142d90 2024-03-25T19:19:49.860 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.858+0000 7f5c57fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:49.860 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.858+0000 7f5c57fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:19:49.860 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.858+0000 7f5c57fff640 10 monclient(hunting): dump: 2024-03-25T19:19:49.860 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 1 2024-03-25T19:19:49.860 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:49.860 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:49.860 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:49.860 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:49.860 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:49.860 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:49.860 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:49.860 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:49.861 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:49.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.861+0000 7f5c57fff640 1 -- 172.21.15.103:0/3648670232 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3203234343 0 0) 0x7f5c70141aa0 con 0x7f5c70142d90 2024-03-25T19:19:49.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.861+0000 7f5c57fff640 1 -- 172.21.15.103:0/3648670232 >> v1:172.21.15.83:6790/0 conn(0x7f5c7006c6b0 legacy=0x7f5c70141390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:49.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.861+0000 7f5c57fff640 1 -- 172.21.15.103:0/3648670232 >> v1:172.21.15.103:6789/0 conn(0x7f5c7006c2d0 legacy=0x7f5c70070d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:49.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.861+0000 7f5c57fff640 10 monclient: _finish_hunting 0 2024-03-25T19:19:49.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.861+0000 7f5c57fff640 1 monclient: found mon.a 2024-03-25T19:19:49.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.861+0000 7f5c57fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:19:49.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.861+0000 7f5c57fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:19:49.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.861+0000 7f5c57fff640 1 -- 172.21.15.103:0/3648670232 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c700702b0 con 0x7f5c70142d90 2024-03-25T19:19:49.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.861+0000 7f5c57fff640 10 monclient: _finish_auth 0 2024-03-25T19:19:49.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.861+0000 7f5c57fff640 10 monclient: _check_auth_tickets 2024-03-25T19:19:49.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.861+0000 7f5c57fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:19:49.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.861+0000 7f5c758f6640 5 monclient: authenticate success, global_id 4155 2024-03-25T19:19:49.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.861+0000 7f5c758f6640 10 monclient: _renew_subs 2024-03-25T19:19:49.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.861+0000 7f5c758f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:19:49.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.861+0000 7f5c758f6640 1 -- 172.21.15.103:0/3648670232 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c7013f610 con 0x7f5c70142d90 2024-03-25T19:19:49.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.861+0000 7f5c758f6640 10 client.4155.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:19:49.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.861+0000 7f5c758f6640 10 monclient: _renew_subs 2024-03-25T19:19:49.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.861+0000 7f5c758f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:19:49.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.861+0000 7f5c758f6640 1 -- 172.21.15.103:0/3648670232 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c70144150 con 0x7f5c70142d90 2024-03-25T19:19:49.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.861+0000 7f5c758f6640 1 librados: init done 2024-03-25T19:19:49.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.862+0000 7f5c758f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:49.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.862+0000 7f5c758f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:49.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.862+0000 7f5c758f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:19:49.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.862+0000 7f5c758f6640 1 -- 172.21.15.103:0/3648670232 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c7013f850 con 0x7f5c70142d90 2024-03-25T19:19:49.961 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.961+0000 7f5c57fff640 1 -- 172.21.15.103:0/3648670232 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c68004b60 con 0x7f5c70142d90 2024-03-25T19:19:49.961 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.961+0000 7f5c57fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:19:49.961 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.961+0000 7f5c57fff640 1 -- 172.21.15.103:0/3648670232 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f5c68002100 con 0x7f5c70142d90 2024-03-25T19:19:49.961 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.961+0000 7f5c57fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:49.962 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.961+0000 7f5c57fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:19:49.962 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.961+0000 7f5c57fff640 10 monclient: dump: 2024-03-25T19:19:49.962 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 1 2024-03-25T19:19:49.962 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:49.962 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:49.962 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:49.962 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:49.962 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:49.962 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:49.962 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:49.962 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:49.962 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:49.962 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.961+0000 7f5c57fff640 1 -- 172.21.15.103:0/3648670232 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f5c680050e0 con 0x7f5c70142d90 2024-03-25T19:19:49.962 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.961+0000 7f5c6d7fa640 4 set_mon_vals no callback set 2024-03-25T19:19:49.963 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.963+0000 7f5c57fff640 1 -- 172.21.15.103:0/3648670232 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 548552571 0 0) 0x7f5c68006910 con 0x7f5c70142d90 2024-03-25T19:19:49.963 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.963+0000 7f5c57fff640 10 client.4155.objecter ms_dispatch 0x7f5c70000b90 osd_map(8..8 src has 1..8) v4 2024-03-25T19:19:49.963 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.963+0000 7f5c57fff640 3 client.4155.objecter handle_osd_map got epochs [8,8] > 0 2024-03-25T19:19:49.963 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.963+0000 7f5c57fff640 3 client.4155.objecter handle_osd_map decoding full epoch 8 2024-03-25T19:19:49.963 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.963+0000 7f5c57fff640 20 client.4155.objecter dump_active .. 0 homeless 2024-03-25T19:19:49.964 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.963+0000 7f5c57fff640 1 -- 172.21.15.103:0/3648670232 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f5c68006320 con 0x7f5c70142d90 2024-03-25T19:19:49.964 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.963+0000 7f5c57fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:49.964 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:49.963+0000 7f5c57fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:19:50.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.036+0000 7f5c758f6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "56c6dbb7-33a7-4af1-a6ab-26c25e08f14c", "id": 6}] 2024-03-25T19:19:50.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.036+0000 7f5c758f6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "56c6dbb7-33a7-4af1-a6ab-26c25e08f14c", "id": 6}] 2024-03-25T19:19:50.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.036+0000 7f5c758f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:19:50.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.036+0000 7f5c758f6640 1 -- 172.21.15.103:0/3648670232 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd new", "uuid": "56c6dbb7-33a7-4af1-a6ab-26c25e08f14c", "id": 6} v 0) v1 -- 0x7f5c70144400 con 0x7f5c70142d90 2024-03-25T19:19:50.062 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.062+0000 7f5c57fff640 1 -- 172.21.15.103:0/3648670232 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "56c6dbb7-33a7-4af1-a6ab-26c25e08f14c", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 2536375704 0 1113811849) 0x7f5c68006610 con 0x7f5c70142d90 2024-03-25T19:19:50.062 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.062+0000 7f5c57fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "56c6dbb7-33a7-4af1-a6ab-26c25e08f14c", "id": 6}] 2024-03-25T19:19:50.062 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.062+0000 7f5c57fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:19:50.062 INFO:teuthology.orchestra.run.smithi103.stdout:6 2024-03-25T19:19:50.063 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.063+0000 7f5c758f6640 10 librados: watch_flush enter 2024-03-25T19:19:50.063 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.063+0000 7f5c758f6640 10 librados: watch_flush exit 2024-03-25T19:19:50.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.063+0000 7f5c758f6640 20 client.4155.objecter shutdown clearing up homeless session... 2024-03-25T19:19:50.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.063+0000 7f5c758f6640 10 client.4155.objecter successfully canceled tick 2024-03-25T19:19:50.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.063+0000 7f5c758f6640 10 monclient: shutdown 2024-03-25T19:19:50.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.063+0000 7f5c758f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:19:50.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.063+0000 7f5c758f6640 1 -- 172.21.15.103:0/3648670232 >> v1:172.21.15.83:6789/0 conn(0x7f5c70142d90 legacy=0x7f5c701e1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:50.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.064+0000 7f5c758f6640 1 -- 172.21.15.103:0/3648670232 shutdown_connections 2024-03-25T19:19:50.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.064+0000 7f5c758f6640 1 -- 172.21.15.103:0/3648670232 >> 172.21.15.103:0/3648670232 conn(0x7f5c700b5e40 msgr2=0x7f5c700b6220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:19:50.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.064+0000 7f5c758f6640 1 -- 172.21.15.103:0/3648670232 shutdown_connections 2024-03-25T19:19:50.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.064+0000 7f5c758f6640 1 -- 172.21.15.103:0/3648670232 wait complete. 2024-03-25T19:19:50.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.064+0000 7f5c758f6640 1 librados: shutdown 2024-03-25T19:19:50.074 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new 612c69a8-2abe-482c-b05c-8909132855ff 7 2024-03-25T19:19:50.136 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-03-25T19:19:50.136+0000 7f02a242b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T19:19:50.174 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.174+0000 7f029d074640 10 monclient: get_monmap_and_config 2024-03-25T19:19:50.174 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.174+0000 7f029d074640 10 monclient: build_initial_monmap 2024-03-25T19:19:50.174 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.174+0000 7f029d074640 10 monclient: monmap: 2024-03-25T19:19:50.174 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 0 2024-03-25T19:19:50.174 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:50.174 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:19:50.175322+0000 2024-03-25T19:19:50.174 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:19:50.175322+0000 2024-03-25T19:19:50.174 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 0 (unknown) 2024-03-25T19:19:50.174 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:50.174 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:19:50.174 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:19:50.174 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:19:50.174 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:50.176 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.175+0000 7f029d074640 1 Processor -- start 2024-03-25T19:19:50.176 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.175+0000 7f029d074640 1 -- start start 2024-03-25T19:19:50.176 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.175+0000 7f029d074640 10 monclient: init 2024-03-25T19:19:50.176 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.175+0000 7f029d074640 10 monclient: _reopen_session rank -1 2024-03-25T19:19:50.176 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.175+0000 7f029d074640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:19:50.176 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.175+0000 7f029d074640 10 monclient(hunting): picked mon.noname-b con 0x7f02980b4700 addr v1:172.21.15.83:6790/0 2024-03-25T19:19:50.176 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.175+0000 7f029d074640 10 monclient(hunting): picked mon.noname-c con 0x7f0298142a20 addr v1:172.21.15.103:6789/0 2024-03-25T19:19:50.176 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.175+0000 7f029d074640 10 monclient(hunting): picked mon.noname-a con 0x7f02980a96b0 addr v1:172.21.15.83:6789/0 2024-03-25T19:19:50.176 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.175+0000 7f029d074640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02980a9d60 con 0x7f02980a96b0 2024-03-25T19:19:50.176 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.175+0000 7f029d074640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f029813ff10 con 0x7f0298142a20 2024-03-25T19:19:50.176 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.175+0000 7f029d074640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02980aa1f0 con 0x7f02980b4700 2024-03-25T19:19:50.176 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.175+0000 7f029d074640 10 monclient(hunting): _renew_subs 2024-03-25T19:19:50.176 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.175+0000 7f029d074640 10 monclient(hunting): authenticate will time out at 6027.310573s 2024-03-25T19:19:50.176 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.176+0000 7f029659c640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f0298142a20 0x7f029813f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.103:33466/0 (socket says 172.21.15.103:33466) 2024-03-25T19:19:50.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.176+0000 7f029659c640 1 -- 172.21.15.103:0/3889265404 learned_addr learned my addr 172.21.15.103:0/3889265404 (peer_addr_for_me v1:172.21.15.103:0/0) 2024-03-25T19:19:50.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.179+0000 7f0295d9b640 1 -- 172.21.15.103:0/3889265404 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 581009081 0 0) 0x7f02980a9d60 con 0x7f02980a96b0 2024-03-25T19:19:50.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.179+0000 7f0295d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:19:50.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.179+0000 7f0295d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:50.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.179+0000 7f0295d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:50.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.179+0000 7f0295d9b640 10 monclient(hunting): my global_id is 4158 2024-03-25T19:19:50.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.179+0000 7f0295d9b640 1 -- 172.21.15.103:0/3889265404 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0280003520 con 0x7f02980a96b0 2024-03-25T19:19:50.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.179+0000 7f0295d9b640 1 -- 172.21.15.103:0/3889265404 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3734483544 0 0) 0x7f02980aa1f0 con 0x7f02980b4700 2024-03-25T19:19:50.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.179+0000 7f0295d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:19:50.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.179+0000 7f0295d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:50.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.179+0000 7f0295d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:50.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.179+0000 7f0295d9b640 10 monclient(hunting): my global_id is 4160 2024-03-25T19:19:50.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.179+0000 7f0295d9b640 1 -- 172.21.15.103:0/3889265404 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02980a9d60 con 0x7f02980b4700 2024-03-25T19:19:50.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.179+0000 7f0295d9b640 1 -- 172.21.15.103:0/3889265404 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4101211774 0 0) 0x7f029813ff10 con 0x7f0298142a20 2024-03-25T19:19:50.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.179+0000 7f0295d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:19:50.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.179+0000 7f0295d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:50.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.179+0000 7f0295d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:50.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.179+0000 7f0295d9b640 10 monclient(hunting): my global_id is 4159 2024-03-25T19:19:50.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.179+0000 7f0295d9b640 1 -- 172.21.15.103:0/3889265404 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02980aa1f0 con 0x7f0298142a20 2024-03-25T19:19:50.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.182+0000 7f0295d9b640 1 -- 172.21.15.103:0/3889265404 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 571312507 0 0) 0x7f0280003520 con 0x7f02980a96b0 2024-03-25T19:19:50.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.182+0000 7f0295d9b640 1 -- 172.21.15.103:0/3889265404 >> v1:172.21.15.83:6790/0 conn(0x7f02980b4700 legacy=0x7f0298142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:50.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.182+0000 7f0295d9b640 1 -- 172.21.15.103:0/3889265404 >> v1:172.21.15.103:6789/0 conn(0x7f0298142a20 legacy=0x7f029813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:50.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.182+0000 7f0295d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:19:50.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.182+0000 7f0295d9b640 1 monclient: found mon.noname-a 2024-03-25T19:19:50.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.182+0000 7f0295d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:19:50.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.182+0000 7f0295d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.83:6789/0 2024-03-25T19:19:50.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.182+0000 7f0295d9b640 1 -- 172.21.15.103:0/3889265404 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02980b4ae0 con 0x7f02980a96b0 2024-03-25T19:19:50.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.182+0000 7f0295d9b640 10 monclient: _finish_auth 0 2024-03-25T19:19:50.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.182+0000 7f0295d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:19:50.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.182+0000 7f0295d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:19:50.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.182+0000 7f029d074640 5 monclient: authenticate success, global_id 4158 2024-03-25T19:19:50.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.182+0000 7f029d074640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:19:50.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.182+0000 7f0295d9b640 1 -- 172.21.15.103:0/3889265404 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f02900019d0 con 0x7f02980a96b0 2024-03-25T19:19:50.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.182+0000 7f0295d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:50.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.182+0000 7f0295d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:19:50.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.182+0000 7f0295d9b640 10 monclient: dump: 2024-03-25T19:19:50.183 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 1 2024-03-25T19:19:50.183 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:50.183 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:50.183 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:50.183 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:50.183 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:50.183 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:50.183 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:50.184 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:50.184 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:50.184 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.182+0000 7f029d074640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:19:50.186 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.187+0000 7f0295d9b640 1 -- 172.21.15.103:0/3889265404 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0290001c20 con 0x7f02980a96b0 2024-03-25T19:19:50.186 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.187+0000 7f0295d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:19:50.186 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.187+0000 7f0295d9b640 1 -- 172.21.15.103:0/3889265404 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f0290001fb0 con 0x7f02980a96b0 2024-03-25T19:19:50.187 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.187+0000 7f0295d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:50.187 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.187+0000 7f0295d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:19:50.187 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.187+0000 7f0295d9b640 10 monclient: dump: 2024-03-25T19:19:50.187 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 1 2024-03-25T19:19:50.187 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:50.187 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:50.187 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:50.187 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:50.187 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:50.187 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:50.187 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:50.187 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:50.187 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:50.187 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.187+0000 7f029d074640 10 monclient: get_monmap_and_config success 2024-03-25T19:19:50.187 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.187+0000 7f029d074640 4 set_mon_vals no callback set 2024-03-25T19:19:50.190 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.188+0000 7f029d074640 10 monclient: shutdown 2024-03-25T19:19:50.190 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.188+0000 7f029d074640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:19:50.190 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.188+0000 7f029d074640 1 -- 172.21.15.103:0/3889265404 >> v1:172.21.15.83:6789/0 conn(0x7f02980a96b0 legacy=0x7f029813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:50.190 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.188+0000 7f029d074640 1 -- 172.21.15.103:0/3889265404 shutdown_connections 2024-03-25T19:19:50.190 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.188+0000 7f029d074640 1 -- 172.21.15.103:0/3889265404 >> 172.21.15.103:0/3889265404 conn(0x7f02980668e0 msgr2=0x7f0298066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:19:50.190 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.188+0000 7f029d074640 1 -- 172.21.15.103:0/3889265404 shutdown_connections 2024-03-25T19:19:50.190 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.188+0000 7f029d074640 1 -- 172.21.15.103:0/3889265404 wait complete. 2024-03-25T19:19:50.191 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.189+0000 7f029d074640 10 monclient: build_initial_monmap 2024-03-25T19:19:50.191 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.189+0000 7f029d074640 10 monclient: monmap: 2024-03-25T19:19:50.191 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 0 2024-03-25T19:19:50.191 INFO:teuthology.orchestra.run.smithi103.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:19:50.191 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 0.000000 2024-03-25T19:19:50.191 INFO:teuthology.orchestra.run.smithi103.stderr:created 0.000000 2024-03-25T19:19:50.191 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 0 (unknown) 2024-03-25T19:19:50.191 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 1 2024-03-25T19:19:50.191 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:19:50.191 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:19:50.191 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:19:50.191 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:50.191 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.189+0000 7f029d074640 1 librados: starting msgr at 2024-03-25T19:19:50.191 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.189+0000 7f029d074640 1 librados: starting objecter 2024-03-25T19:19:50.191 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.189+0000 7f029d074640 1 Processor -- start 2024-03-25T19:19:50.192 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.189+0000 7f029d074640 1 -- start start 2024-03-25T19:19:50.192 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.189+0000 7f029d074640 1 librados: setting wanted keys 2024-03-25T19:19:50.192 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.189+0000 7f029d074640 1 librados: calling monclient init 2024-03-25T19:19:50.192 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.189+0000 7f029d074640 10 monclient: init 2024-03-25T19:19:50.192 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.190+0000 7f029d074640 10 monclient: _reopen_session rank -1 2024-03-25T19:19:50.192 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.190+0000 7f029d074640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:19:50.192 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.190+0000 7f029d074640 10 monclient(hunting): picked mon.noname-a con 0x7f02980a96b0 addr v1:172.21.15.83:6789/0 2024-03-25T19:19:50.192 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.190+0000 7f029d074640 10 monclient(hunting): picked mon.noname-b con 0x7f02980b4700 addr v1:172.21.15.103:6789/0 2024-03-25T19:19:50.192 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.190+0000 7f029d074640 10 monclient(hunting): picked mon.noname-c con 0x7f0298142a20 addr v1:172.21.15.83:6790/0 2024-03-25T19:19:50.192 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.190+0000 7f029d074640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f029813ace0 con 0x7f02980a96b0 2024-03-25T19:19:50.192 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.190+0000 7f029d074640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02980abb70 con 0x7f02980b4700 2024-03-25T19:19:50.192 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.190+0000 7f029d074640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02980abd00 con 0x7f0298142a20 2024-03-25T19:19:50.192 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.190+0000 7f029d074640 10 monclient(hunting): _renew_subs 2024-03-25T19:19:50.192 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.190+0000 7f029d074640 10 monclient(hunting): authenticate will time out at 6027.324804s 2024-03-25T19:19:50.192 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.190+0000 7f0296d9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f02980a96b0 0x7f02980ac760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.103:42440/0 (socket says 172.21.15.103:42440) 2024-03-25T19:19:50.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.190+0000 7f0296d9d640 1 -- 172.21.15.103:0/863672441 learned_addr learned my addr 172.21.15.103:0/863672441 (peer_addr_for_me v1:172.21.15.103:0/0) 2024-03-25T19:19:50.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.190+0000 7f027f7fe640 10 client.?.objecter ms_handle_connect 0x7f02980a96b0 2024-03-25T19:19:50.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.190+0000 7f027f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:50.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.190+0000 7f027f7fe640 10 client.?.objecter ms_handle_connect 0x7f0298142a20 2024-03-25T19:19:50.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.191+0000 7f027f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:50.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.191+0000 7f027f7fe640 10 client.?.objecter ms_handle_connect 0x7f02980b4700 2024-03-25T19:19:50.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.191+0000 7f027f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:50.194 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.192+0000 7f027f7fe640 1 -- 172.21.15.103:0/863672441 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1082872418 0 0) 0x7f029813ace0 con 0x7f02980a96b0 2024-03-25T19:19:50.194 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.192+0000 7f027f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:19:50.194 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.192+0000 7f027f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:50.194 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.192+0000 7f027f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:50.194 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.192+0000 7f027f7fe640 10 monclient(hunting): my global_id is 4161 2024-03-25T19:19:50.194 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.192+0000 7f027f7fe640 1 -- 172.21.15.103:0/863672441 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0284004eb0 con 0x7f02980a96b0 2024-03-25T19:19:50.194 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.193+0000 7f027f7fe640 1 -- 172.21.15.103:0/863672441 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3062900260 0 0) 0x7f02980abb70 con 0x7f02980b4700 2024-03-25T19:19:50.194 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.193+0000 7f027f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:19:50.195 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.193+0000 7f027f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:50.195 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.193+0000 7f027f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:50.195 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.193+0000 7f027f7fe640 10 monclient(hunting): my global_id is 4162 2024-03-25T19:19:50.195 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.193+0000 7f027f7fe640 1 -- 172.21.15.103:0/863672441 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f029813ace0 con 0x7f02980b4700 2024-03-25T19:19:50.195 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.193+0000 7f027f7fe640 1 -- 172.21.15.103:0/863672441 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2851129195 0 0) 0x7f02980abd00 con 0x7f0298142a20 2024-03-25T19:19:50.195 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.193+0000 7f027f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:19:50.195 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.193+0000 7f027f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:50.195 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.193+0000 7f027f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:50.195 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.193+0000 7f027f7fe640 10 monclient(hunting): my global_id is 4163 2024-03-25T19:19:50.195 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.193+0000 7f027f7fe640 1 -- 172.21.15.103:0/863672441 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02980abb70 con 0x7f0298142a20 2024-03-25T19:19:50.197 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.197+0000 7f027f7fe640 1 -- 172.21.15.103:0/863672441 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3517788771 0 0) 0x7f0284004eb0 con 0x7f02980a96b0 2024-03-25T19:19:50.197 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.197+0000 7f027f7fe640 1 -- 172.21.15.103:0/863672441 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02980abd00 con 0x7f02980a96b0 2024-03-25T19:19:50.197 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.197+0000 7f027f7fe640 1 -- 172.21.15.103:0/863672441 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f02880019b0 con 0x7f02980a96b0 2024-03-25T19:19:50.197 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.197+0000 7f027f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:50.197 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.197+0000 7f027f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:19:50.197 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.197+0000 7f027f7fe640 10 monclient(hunting): dump: 2024-03-25T19:19:50.197 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 1 2024-03-25T19:19:50.197 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:50.197 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:50.197 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:50.197 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:50.198 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:50.198 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:50.198 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:50.198 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:50.198 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:50.198 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.197+0000 7f027f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:19:50.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.199+0000 7f027f7fe640 1 -- 172.21.15.103:0/863672441 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3358514883 0 0) 0x7f02980abb70 con 0x7f0298142a20 2024-03-25T19:19:50.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.199+0000 7f027f7fe640 1 -- 172.21.15.103:0/863672441 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0284004eb0 con 0x7f0298142a20 2024-03-25T19:19:50.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.199+0000 7f027f7fe640 1 -- 172.21.15.103:0/863672441 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f0290001f30 con 0x7f0298142a20 2024-03-25T19:19:50.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.199+0000 7f027f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:50.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.199+0000 7f027f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:19:50.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.199+0000 7f027f7fe640 10 monclient(hunting): dump: 2024-03-25T19:19:50.199 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 1 2024-03-25T19:19:50.199 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:50.199 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:50.200 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:50.200 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:50.200 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:50.200 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:50.200 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:50.200 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:50.200 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:50.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.200+0000 7f027f7fe640 1 -- 172.21.15.103:0/863672441 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 952872734 0 0) 0x7f029813ace0 con 0x7f02980b4700 2024-03-25T19:19:50.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.200+0000 7f027f7fe640 1 -- 172.21.15.103:0/863672441 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02980abb70 con 0x7f02980b4700 2024-03-25T19:19:50.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.200+0000 7f027f7fe640 1 -- 172.21.15.103:0/863672441 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f028c0018f0 con 0x7f02980b4700 2024-03-25T19:19:50.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.200+0000 7f027f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:50.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.200+0000 7f027f7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:19:50.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.200+0000 7f027f7fe640 10 monclient(hunting): dump: 2024-03-25T19:19:50.200 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 1 2024-03-25T19:19:50.200 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:50.200 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:50.201 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:50.201 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:50.201 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:50.201 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:50.201 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:50.201 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:50.201 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:50.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.201+0000 7f027f7fe640 1 -- 172.21.15.103:0/863672441 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3404187892 0 0) 0x7f02980abd00 con 0x7f02980a96b0 2024-03-25T19:19:50.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.201+0000 7f027f7fe640 1 -- 172.21.15.103:0/863672441 >> v1:172.21.15.83:6790/0 conn(0x7f0298142a20 legacy=0x7f02981e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:50.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.201+0000 7f027f7fe640 1 -- 172.21.15.103:0/863672441 >> v1:172.21.15.103:6789/0 conn(0x7f02980b4700 legacy=0x7f02980ae5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:50.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.201+0000 7f027f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:19:50.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.201+0000 7f027f7fe640 1 monclient: found mon.a 2024-03-25T19:19:50.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.201+0000 7f027f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:19:50.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.201+0000 7f027f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:19:50.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.201+0000 7f027f7fe640 1 -- 172.21.15.103:0/863672441 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02980ace70 con 0x7f02980a96b0 2024-03-25T19:19:50.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.201+0000 7f027f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:19:50.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.201+0000 7f027f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:19:50.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.201+0000 7f027f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:19:50.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.201+0000 7f029d074640 5 monclient: authenticate success, global_id 4161 2024-03-25T19:19:50.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.201+0000 7f029d074640 10 monclient: _renew_subs 2024-03-25T19:19:50.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.201+0000 7f029d074640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:19:50.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.201+0000 7f029d074640 1 -- 172.21.15.103:0/863672441 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02980ad0c0 con 0x7f02980a96b0 2024-03-25T19:19:50.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.201+0000 7f029d074640 10 client.4161.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:19:50.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.201+0000 7f029d074640 10 monclient: _renew_subs 2024-03-25T19:19:50.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.201+0000 7f029d074640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:19:50.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.201+0000 7f029d074640 1 -- 172.21.15.103:0/863672441 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f02980aeee0 con 0x7f02980a96b0 2024-03-25T19:19:50.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.201+0000 7f029d074640 1 librados: init done 2024-03-25T19:19:50.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.202+0000 7f029d074640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:50.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.202+0000 7f029d074640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:50.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.202+0000 7f029d074640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:19:50.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.202+0000 7f029d074640 1 -- 172.21.15.103:0/863672441 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f029804f730 con 0x7f02980a96b0 2024-03-25T19:19:50.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.299+0000 7f027f7fe640 1 -- 172.21.15.103:0/863672441 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0288003610 con 0x7f02980a96b0 2024-03-25T19:19:50.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.299+0000 7f027f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:19:50.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.299+0000 7f027f7fe640 1 -- 172.21.15.103:0/863672441 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f02880039a0 con 0x7f02980a96b0 2024-03-25T19:19:50.300 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.299+0000 7f027f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:50.300 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.299+0000 7f027f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:19:50.300 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.300+0000 7f027f7fe640 10 monclient: dump: 2024-03-25T19:19:50.300 INFO:teuthology.orchestra.run.smithi103.stderr:epoch 1 2024-03-25T19:19:50.300 INFO:teuthology.orchestra.run.smithi103.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:50.300 INFO:teuthology.orchestra.run.smithi103.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:50.300 INFO:teuthology.orchestra.run.smithi103.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:50.300 INFO:teuthology.orchestra.run.smithi103.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:50.300 INFO:teuthology.orchestra.run.smithi103.stderr:election_strategy: 3 2024-03-25T19:19:50.300 INFO:teuthology.orchestra.run.smithi103.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:50.300 INFO:teuthology.orchestra.run.smithi103.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:50.300 INFO:teuthology.orchestra.run.smithi103.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:50.300 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:19:50.300 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.300+0000 7f027f7fe640 1 -- 172.21.15.103:0/863672441 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f0288003f50 con 0x7f02980a96b0 2024-03-25T19:19:50.300 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.300+0000 7f027f7fe640 1 -- 172.21.15.103:0/863672441 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 3088755673 0 0) 0x7f02880067e0 con 0x7f02980a96b0 2024-03-25T19:19:50.300 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.300+0000 7f027f7fe640 10 client.4161.objecter ms_dispatch 0x7f0298000b90 osd_map(9..9 src has 1..9) v4 2024-03-25T19:19:50.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.300+0000 7f027f7fe640 3 client.4161.objecter handle_osd_map got epochs [9,9] > 0 2024-03-25T19:19:50.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.300+0000 7f027f7fe640 3 client.4161.objecter handle_osd_map decoding full epoch 9 2024-03-25T19:19:50.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.300+0000 7f029559a640 4 set_mon_vals no callback set 2024-03-25T19:19:50.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.300+0000 7f027f7fe640 20 client.4161.objecter dump_active .. 0 homeless 2024-03-25T19:19:50.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.300+0000 7f027f7fe640 1 -- 172.21.15.103:0/863672441 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f0288004200 con 0x7f02980a96b0 2024-03-25T19:19:50.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.300+0000 7f027f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:50.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.300+0000 7f027f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:19:50.372 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.372+0000 7f029d074640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "612c69a8-2abe-482c-b05c-8909132855ff", "id": 7}] 2024-03-25T19:19:50.372 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.372+0000 7f029d074640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "612c69a8-2abe-482c-b05c-8909132855ff", "id": 7}] 2024-03-25T19:19:50.372 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.372+0000 7f029d074640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:19:50.372 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.372+0000 7f029d074640 1 -- 172.21.15.103:0/863672441 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd new", "uuid": "612c69a8-2abe-482c-b05c-8909132855ff", "id": 7} v 0) v1 -- 0x7f0298136a00 con 0x7f02980a96b0 2024-03-25T19:19:50.398 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.399+0000 7f027f7fe640 1 -- 172.21.15.103:0/863672441 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "612c69a8-2abe-482c-b05c-8909132855ff", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 1150957813 0 1371665406) 0x7f0288006230 con 0x7f02980a96b0 2024-03-25T19:19:50.398 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.399+0000 7f027f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "612c69a8-2abe-482c-b05c-8909132855ff", "id": 7}] 2024-03-25T19:19:50.398 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.399+0000 7f027f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:19:50.399 INFO:teuthology.orchestra.run.smithi103.stdout:7 2024-03-25T19:19:50.400 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.400+0000 7f029d074640 10 librados: watch_flush enter 2024-03-25T19:19:50.400 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.400+0000 7f029d074640 10 librados: watch_flush exit 2024-03-25T19:19:50.400 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.400+0000 7f029d074640 20 client.4161.objecter shutdown clearing up homeless session... 2024-03-25T19:19:50.400 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.400+0000 7f029d074640 10 client.4161.objecter successfully canceled tick 2024-03-25T19:19:50.400 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.400+0000 7f029d074640 10 monclient: shutdown 2024-03-25T19:19:50.400 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.400+0000 7f029d074640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:19:50.401 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.400+0000 7f029d074640 1 -- 172.21.15.103:0/863672441 >> v1:172.21.15.83:6789/0 conn(0x7f02980a96b0 legacy=0x7f02980ac760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:50.401 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.400+0000 7f029d074640 1 -- 172.21.15.103:0/863672441 shutdown_connections 2024-03-25T19:19:50.401 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.400+0000 7f029d074640 1 -- 172.21.15.103:0/863672441 >> 172.21.15.103:0/863672441 conn(0x7f02980668e0 msgr2=0x7f02980b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:19:50.401 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.400+0000 7f029d074640 1 -- 172.21.15.103:0/863672441 shutdown_connections 2024-03-25T19:19:50.401 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.400+0000 7f029d074640 1 -- 172.21.15.103:0/863672441 wait complete. 2024-03-25T19:19:50.401 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-25T19:19:50.400+0000 7f029d074640 1 librados: shutdown 2024-03-25T19:19:50.410 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:19:50.410 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-25T19:19:50.411 DEBUG:teuthology.orchestra.run.smithi083:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-03-25T19:19:50.413 INFO:tasks.ceph.osd.0:Started 2024-03-25T19:19:50.413 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:19:50.413 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-25T19:19:50.413 DEBUG:teuthology.orchestra.run.smithi083:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-03-25T19:19:50.415 INFO:tasks.ceph.osd.1:Started 2024-03-25T19:19:50.416 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:19:50.416 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-25T19:19:50.416 DEBUG:teuthology.orchestra.run.smithi083:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-03-25T19:19:50.418 INFO:tasks.ceph.osd.2:Started 2024-03-25T19:19:50.418 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:19:50.418 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-25T19:19:50.418 DEBUG:teuthology.orchestra.run.smithi083:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-03-25T19:19:50.420 INFO:tasks.ceph.osd.3:Started 2024-03-25T19:19:50.421 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:19:50.421 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-25T19:19:50.421 DEBUG:teuthology.orchestra.run.smithi103:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-03-25T19:19:50.423 INFO:tasks.ceph.osd.4:Started 2024-03-25T19:19:50.423 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:19:50.424 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-25T19:19:50.424 DEBUG:teuthology.orchestra.run.smithi103:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-03-25T19:19:50.426 INFO:tasks.ceph.osd.5:Started 2024-03-25T19:19:50.426 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:19:50.426 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-25T19:19:50.426 DEBUG:teuthology.orchestra.run.smithi103:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-03-25T19:19:50.428 INFO:tasks.ceph.osd.6:Started 2024-03-25T19:19:50.428 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:19:50.429 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-25T19:19:50.429 DEBUG:teuthology.orchestra.run.smithi103:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-03-25T19:19:50.432 INFO:tasks.ceph.osd.7:Started 2024-03-25T19:19:50.432 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T19:19:50.446 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-03-25T19:19:50.446+0000 7f02a242b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T19:19:50.505 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-25T19:19:50.505+0000 7f54aa63c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T19:19:50.527 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-03-25T19:19:50.527+0000 7f02a242b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T19:19:50.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.584+0000 7f15f0ef6640 10 monclient: get_monmap_and_config 2024-03-25T19:19:50.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.584+0000 7f15f0ef6640 10 monclient: build_initial_monmap 2024-03-25T19:19:50.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.584+0000 7f15f0ef6640 10 monclient: monmap: 2024-03-25T19:19:50.584 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T19:19:50.584 INFO:teuthology.orchestra.run.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:50.584 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T19:19:50.585192+0000 2024-03-25T19:19:50.584 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T19:19:50.585192+0000 2024-03-25T19:19:50.585 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:19:50.585 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 3 2024-03-25T19:19:50.585 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:19:50.585 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:19:50.585 INFO:teuthology.orchestra.run.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:19:50.585 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T19:19:50.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.584+0000 7f15f0ef6640 1 Processor -- start 2024-03-25T19:19:50.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.585+0000 7f15f0ef6640 1 -- start start 2024-03-25T19:19:50.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.585+0000 7f15f0ef6640 10 monclient: init 2024-03-25T19:19:50.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.585+0000 7f15f0ef6640 10 monclient: _reopen_session rank -1 2024-03-25T19:19:50.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.585+0000 7f15f0ef6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:19:50.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.585+0000 7f15f0ef6640 10 monclient(hunting): picked mon.noname-b con 0x7f15ec0b4700 addr v1:172.21.15.83:6790/0 2024-03-25T19:19:50.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.585+0000 7f15f0ef6640 10 monclient(hunting): picked mon.noname-c con 0x7f15ec142a20 addr v1:172.21.15.103:6789/0 2024-03-25T19:19:50.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.585+0000 7f15f0ef6640 10 monclient(hunting): picked mon.noname-a con 0x7f15ec0a96b0 addr v1:172.21.15.83:6789/0 2024-03-25T19:19:50.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.585+0000 7f15f0ef6640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15ec0a9d60 con 0x7f15ec0a96b0 2024-03-25T19:19:50.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.585+0000 7f15f0ef6640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15ec13ff10 con 0x7f15ec142a20 2024-03-25T19:19:50.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.586+0000 7f15f0ef6640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15ec0aa1f0 con 0x7f15ec0b4700 2024-03-25T19:19:50.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.586+0000 7f15f0ef6640 10 monclient(hunting): _renew_subs 2024-03-25T19:19:50.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.586+0000 7f15f0ef6640 10 monclient(hunting): authenticate will time out at 6047.858100s 2024-03-25T19:19:50.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.586+0000 7f15ea59c640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f15ec0b4700 0x7f15ec142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:38968/0 (socket says 172.21.15.83:38968) 2024-03-25T19:19:50.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.586+0000 7f15ea59c640 1 -- 172.21.15.83:0/4019042366 learned_addr learned my addr 172.21.15.83:0/4019042366 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:19:50.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.589+0000 7f15e959a640 1 -- 172.21.15.83:0/4019042366 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 359527588 0 0) 0x7f15ec0aa1f0 con 0x7f15ec0b4700 2024-03-25T19:19:50.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.589+0000 7f15e959a640 10 monclient(hunting): _negotiate 2024-03-25T19:19:50.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.589+0000 7f15e959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:50.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.589+0000 7f15e959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:50.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.589+0000 7f15e959a640 10 monclient(hunting): my global_id is 4166 2024-03-25T19:19:50.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.589+0000 7f15e959a640 1 -- 172.21.15.83:0/4019042366 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15d4003520 con 0x7f15ec0b4700 2024-03-25T19:19:50.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.589+0000 7f15e959a640 1 -- 172.21.15.83:0/4019042366 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1935638826 0 0) 0x7f15ec13ff10 con 0x7f15ec142a20 2024-03-25T19:19:50.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.589+0000 7f15e959a640 10 monclient(hunting): _negotiate 2024-03-25T19:19:50.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.589+0000 7f15e959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:50.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.589+0000 7f15e959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:50.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.589+0000 7f15e959a640 10 monclient(hunting): my global_id is 4165 2024-03-25T19:19:50.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.589+0000 7f15e959a640 1 -- 172.21.15.83:0/4019042366 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15d4003840 con 0x7f15ec142a20 2024-03-25T19:19:50.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.590+0000 7f15e959a640 1 -- 172.21.15.83:0/4019042366 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3756227328 0 0) 0x7f15ec0a9d60 con 0x7f15ec0a96b0 2024-03-25T19:19:50.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.590+0000 7f15e959a640 10 monclient(hunting): _negotiate 2024-03-25T19:19:50.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.590+0000 7f15e959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:50.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.590+0000 7f15e959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:50.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.590+0000 7f15e959a640 10 monclient(hunting): my global_id is 4164 2024-03-25T19:19:50.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.590+0000 7f15e959a640 1 -- 172.21.15.83:0/4019042366 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15ec13ff10 con 0x7f15ec0a96b0 2024-03-25T19:19:50.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.592+0000 7f15e959a640 1 -- 172.21.15.83:0/4019042366 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1824627401 0 0) 0x7f15d4003520 con 0x7f15ec0b4700 2024-03-25T19:19:50.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.592+0000 7f15e959a640 1 -- 172.21.15.83:0/4019042366 >> v1:172.21.15.103:6789/0 conn(0x7f15ec142a20 legacy=0x7f15ec13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:50.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.592+0000 7f15e959a640 1 -- 172.21.15.83:0/4019042366 >> v1:172.21.15.83:6789/0 conn(0x7f15ec0a96b0 legacy=0x7f15ec13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:50.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.592+0000 7f15e959a640 10 monclient: _finish_hunting 0 2024-03-25T19:19:50.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.592+0000 7f15e959a640 1 monclient: found mon.noname-b 2024-03-25T19:19:50.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.592+0000 7f15e959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:19:50.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.592+0000 7f15e959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.83:6790/0 2024-03-25T19:19:50.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.592+0000 7f15e959a640 1 -- 172.21.15.83:0/4019042366 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15ec0b4ae0 con 0x7f15ec0b4700 2024-03-25T19:19:50.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.592+0000 7f15e959a640 10 monclient: _finish_auth 0 2024-03-25T19:19:50.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.592+0000 7f15e959a640 10 monclient: _check_auth_tickets 2024-03-25T19:19:50.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.592+0000 7f15e959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:19:50.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.592+0000 7f15e959a640 1 -- 172.21.15.83:0/4019042366 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f15dc001630 con 0x7f15ec0b4700 2024-03-25T19:19:50.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.592+0000 7f15e959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:50.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.592+0000 7f15e959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:19:50.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.592+0000 7f15e959a640 10 monclient: dump: 2024-03-25T19:19:50.599 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T19:19:50.599 INFO:teuthology.orchestra.run.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:50.599 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:50.599 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:50.600 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:50.600 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 3 2024-03-25T19:19:50.600 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:50.600 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:50.600 INFO:teuthology.orchestra.run.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:50.600 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T19:19:50.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.592+0000 7f15f0ef6640 5 monclient: authenticate success, global_id 4166 2024-03-25T19:19:50.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.592+0000 7f15f0ef6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:19:50.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.594+0000 7f15e959a640 1 -- 172.21.15.83:0/4019042366 <== mon.2 v1:172.21.15.83:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15dc0018c0 con 0x7f15ec0b4700 2024-03-25T19:19:50.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.594+0000 7f15e959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:19:50.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.594+0000 7f15f0ef6640 10 monclient: get_monmap_and_config success 2024-03-25T19:19:50.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.594+0000 7f15f0ef6640 4 set_mon_vals no callback set 2024-03-25T19:19:50.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.595+0000 7f15e959a640 1 -- 172.21.15.83:0/4019042366 <== mon.2 v1:172.21.15.83:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f15dc001630 con 0x7f15ec0b4700 2024-03-25T19:19:50.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.595+0000 7f15e959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:50.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.595+0000 7f15e959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:19:50.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.595+0000 7f15e959a640 10 monclient: dump: 2024-03-25T19:19:50.600 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T19:19:50.601 INFO:teuthology.orchestra.run.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:50.601 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:50.601 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:50.601 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:50.601 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 3 2024-03-25T19:19:50.601 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:50.601 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:50.601 INFO:teuthology.orchestra.run.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:50.601 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T19:19:50.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.595+0000 7f15f0ef6640 10 monclient: shutdown 2024-03-25T19:19:50.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.595+0000 7f15f0ef6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:19:50.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.595+0000 7f15f0ef6640 1 -- 172.21.15.83:0/4019042366 >> v1:172.21.15.83:6790/0 conn(0x7f15ec0b4700 legacy=0x7f15ec142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:50.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.595+0000 7f15f0ef6640 1 -- 172.21.15.83:0/4019042366 shutdown_connections 2024-03-25T19:19:50.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.595+0000 7f15f0ef6640 1 -- 172.21.15.83:0/4019042366 >> 172.21.15.83:0/4019042366 conn(0x7f15ec0668e0 msgr2=0x7f15ec066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:19:50.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.596+0000 7f15f0ef6640 1 -- 172.21.15.83:0/4019042366 shutdown_connections 2024-03-25T19:19:50.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.596+0000 7f15f0ef6640 1 -- 172.21.15.83:0/4019042366 wait complete. 2024-03-25T19:19:50.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.597+0000 7f15f0ef6640 10 monclient: build_initial_monmap 2024-03-25T19:19:50.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.597+0000 7f15f0ef6640 10 monclient: monmap: 2024-03-25T19:19:50.602 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T19:19:50.602 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:19:50.602 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T19:19:50.602 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T19:19:50.602 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:19:50.602 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T19:19:50.602 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:19:50.602 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:19:50.602 INFO:teuthology.orchestra.run.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:19:50.602 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T19:19:50.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.597+0000 7f15f0ef6640 1 librados: starting msgr at 2024-03-25T19:19:50.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.597+0000 7f15f0ef6640 1 librados: starting objecter 2024-03-25T19:19:50.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.597+0000 7f15f0ef6640 1 Processor -- start 2024-03-25T19:19:50.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.597+0000 7f15f0ef6640 1 -- start start 2024-03-25T19:19:50.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.597+0000 7f15f0ef6640 1 librados: setting wanted keys 2024-03-25T19:19:50.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.597+0000 7f15f0ef6640 1 librados: calling monclient init 2024-03-25T19:19:50.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.597+0000 7f15f0ef6640 10 monclient: init 2024-03-25T19:19:50.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.598+0000 7f15f0ef6640 10 monclient: _reopen_session rank -1 2024-03-25T19:19:50.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.598+0000 7f15f0ef6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:19:50.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.598+0000 7f15f0ef6640 10 monclient(hunting): picked mon.noname-c con 0x7f15ec0a96b0 addr v1:172.21.15.83:6790/0 2024-03-25T19:19:50.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.598+0000 7f15f0ef6640 10 monclient(hunting): picked mon.noname-a con 0x7f15ec142a20 addr v1:172.21.15.83:6789/0 2024-03-25T19:19:50.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.598+0000 7f15f0ef6640 10 monclient(hunting): picked mon.noname-b con 0x7f15ec14be20 addr v1:172.21.15.103:6789/0 2024-03-25T19:19:50.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.598+0000 7f15f0ef6640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15ec133e30 con 0x7f15ec142a20 2024-03-25T19:19:50.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.598+0000 7f15f0ef6640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15ec121520 con 0x7f15ec14be20 2024-03-25T19:19:50.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.598+0000 7f15f0ef6640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15ec13cc30 con 0x7f15ec0a96b0 2024-03-25T19:19:50.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.599+0000 7f15f0ef6640 10 monclient(hunting): _renew_subs 2024-03-25T19:19:50.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.599+0000 7f15f0ef6640 10 monclient(hunting): authenticate will time out at 6047.870830s 2024-03-25T19:19:50.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.599+0000 7f15ea59c640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f15ec0a96b0 0x7f15ec1342c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:38980/0 (socket says 172.21.15.83:38980) 2024-03-25T19:19:50.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.599+0000 7f15ea59c640 1 -- 172.21.15.83:0/1262036209 learned_addr learned my addr 172.21.15.83:0/1262036209 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:19:50.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.599+0000 7f15daffd640 10 client.?.objecter ms_handle_connect 0x7f15ec0a96b0 2024-03-25T19:19:50.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.599+0000 7f15daffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:50.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.600+0000 7f15daffd640 10 client.?.objecter ms_handle_connect 0x7f15ec142a20 2024-03-25T19:19:50.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.600+0000 7f15daffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:50.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.600+0000 7f15daffd640 10 client.?.objecter ms_handle_connect 0x7f15ec14be20 2024-03-25T19:19:50.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.600+0000 7f15daffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:50.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.602+0000 7f15daffd640 1 -- 172.21.15.83:0/1262036209 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1749093880 0 0) 0x7f15ec133e30 con 0x7f15ec142a20 2024-03-25T19:19:50.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.602+0000 7f15daffd640 10 monclient(hunting): _negotiate 2024-03-25T19:19:50.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.602+0000 7f15daffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:50.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.602+0000 7f15daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:50.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.602+0000 7f15daffd640 10 monclient(hunting): my global_id is 4167 2024-03-25T19:19:50.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.602+0000 7f15daffd640 1 -- 172.21.15.83:0/1262036209 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15cc004eb0 con 0x7f15ec142a20 2024-03-25T19:19:50.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.602+0000 7f15daffd640 1 -- 172.21.15.83:0/1262036209 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 759538163 0 0) 0x7f15ec13cc30 con 0x7f15ec0a96b0 2024-03-25T19:19:50.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.602+0000 7f15daffd640 10 monclient(hunting): _negotiate 2024-03-25T19:19:50.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.602+0000 7f15daffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:50.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.602+0000 7f15daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:50.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.602+0000 7f15daffd640 10 monclient(hunting): my global_id is 4169 2024-03-25T19:19:50.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.602+0000 7f15daffd640 1 -- 172.21.15.83:0/1262036209 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15ec133e30 con 0x7f15ec0a96b0 2024-03-25T19:19:50.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.603+0000 7f15daffd640 1 -- 172.21.15.83:0/1262036209 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2302582999 0 0) 0x7f15ec121520 con 0x7f15ec14be20 2024-03-25T19:19:50.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.603+0000 7f15daffd640 10 monclient(hunting): _negotiate 2024-03-25T19:19:50.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.603+0000 7f15daffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:50.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.603+0000 7f15daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:50.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.603+0000 7f15daffd640 10 monclient(hunting): my global_id is 4168 2024-03-25T19:19:50.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.603+0000 7f15daffd640 1 -- 172.21.15.83:0/1262036209 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15cc005360 con 0x7f15ec14be20 2024-03-25T19:19:50.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.607+0000 7f15daffd640 1 -- 172.21.15.83:0/1262036209 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3835124724 0 0) 0x7f15cc004eb0 con 0x7f15ec142a20 2024-03-25T19:19:50.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.607+0000 7f15daffd640 1 -- 172.21.15.83:0/1262036209 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15ec121520 con 0x7f15ec142a20 2024-03-25T19:19:50.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.607+0000 7f15daffd640 1 -- 172.21.15.83:0/1262036209 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f15e00018b0 con 0x7f15ec142a20 2024-03-25T19:19:50.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.607+0000 7f15daffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:50.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.607+0000 7f15daffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:19:50.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.607+0000 7f15daffd640 10 monclient(hunting): dump: 2024-03-25T19:19:50.610 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T19:19:50.610 INFO:teuthology.orchestra.run.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:50.610 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:50.610 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:50.610 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:50.610 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 3 2024-03-25T19:19:50.610 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:50.610 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:50.610 INFO:teuthology.orchestra.run.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:50.610 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T19:19:50.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.607+0000 7f15daffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:19:50.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.609+0000 7f15daffd640 1 -- 172.21.15.83:0/1262036209 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3151316051 0 0) 0x7f15ec133e30 con 0x7f15ec0a96b0 2024-03-25T19:19:50.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.609+0000 7f15daffd640 1 -- 172.21.15.83:0/1262036209 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15cc004eb0 con 0x7f15ec0a96b0 2024-03-25T19:19:50.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.612+0000 7f15daffd640 1 -- 172.21.15.83:0/1262036209 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1406588605 0 0) 0x7f15cc005360 con 0x7f15ec14be20 2024-03-25T19:19:50.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.612+0000 7f15daffd640 1 -- 172.21.15.83:0/1262036209 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15ec133e30 con 0x7f15ec14be20 2024-03-25T19:19:50.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.612+0000 7f15daffd640 1 -- 172.21.15.83:0/1262036209 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f15e4001c70 con 0x7f15ec14be20 2024-03-25T19:19:50.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.612+0000 7f15daffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:50.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.612+0000 7f15daffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:19:50.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.612+0000 7f15daffd640 10 monclient(hunting): dump: 2024-03-25T19:19:50.612 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T19:19:50.612 INFO:teuthology.orchestra.run.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:50.612 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:50.612 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:50.612 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:50.612 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 3 2024-03-25T19:19:50.612 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:50.612 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:50.612 INFO:teuthology.orchestra.run.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:50.612 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T19:19:50.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.612+0000 7f15daffd640 1 -- 172.21.15.83:0/1262036209 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 26525705 0 0) 0x7f15ec121520 con 0x7f15ec142a20 2024-03-25T19:19:50.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.612+0000 7f15daffd640 1 -- 172.21.15.83:0/1262036209 >> v1:172.21.15.83:6790/0 conn(0x7f15ec0a96b0 legacy=0x7f15ec1342c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:50.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.612+0000 7f15daffd640 1 -- 172.21.15.83:0/1262036209 >> v1:172.21.15.103:6789/0 conn(0x7f15ec14be20 legacy=0x7f15ec1f7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:50.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.612+0000 7f15daffd640 10 monclient: _finish_hunting 0 2024-03-25T19:19:50.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.612+0000 7f15daffd640 1 monclient: found mon.a 2024-03-25T19:19:50.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.612+0000 7f15daffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:19:50.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.612+0000 7f15daffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:19:50.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.612+0000 7f15daffd640 1 -- 172.21.15.83:0/1262036209 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15ec1349d0 con 0x7f15ec142a20 2024-03-25T19:19:50.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.612+0000 7f15daffd640 10 monclient: _finish_auth 0 2024-03-25T19:19:50.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.612+0000 7f15daffd640 10 monclient: _check_auth_tickets 2024-03-25T19:19:50.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.612+0000 7f15daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:19:50.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.612+0000 7f15f0ef6640 5 monclient: authenticate success, global_id 4167 2024-03-25T19:19:50.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.612+0000 7f15f0ef6640 10 monclient: _renew_subs 2024-03-25T19:19:50.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.612+0000 7f15f0ef6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:19:50.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.612+0000 7f15f0ef6640 1 -- 172.21.15.83:0/1262036209 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15ec14e100 con 0x7f15ec142a20 2024-03-25T19:19:50.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.612+0000 7f15f0ef6640 10 client.4167.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:19:50.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.612+0000 7f15f0ef6640 10 monclient: _renew_subs 2024-03-25T19:19:50.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.612+0000 7f15f0ef6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:19:50.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.612+0000 7f15f0ef6640 1 -- 172.21.15.83:0/1262036209 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f15ec120330 con 0x7f15ec142a20 2024-03-25T19:19:50.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.612+0000 7f15f0ef6640 1 librados: init done 2024-03-25T19:19:50.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.614+0000 7f15d8ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:50.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.614+0000 7f15d8ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:50.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.614+0000 7f15d8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:19:50.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.614+0000 7f15d8ff9640 1 -- 172.21.15.83:0/1262036209 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15d0005080 con 0x7f15ec142a20 2024-03-25T19:19:50.655 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-03-25T19:19:50.656+0000 7f02a242b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T19:19:50.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.716+0000 7f15daffd640 1 -- 172.21.15.83:0/1262036209 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15e0001d40 con 0x7f15ec142a20 2024-03-25T19:19:50.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.716+0000 7f15daffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:19:50.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.716+0000 7f15daffd640 1 -- 172.21.15.83:0/1262036209 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f15e0003790 con 0x7f15ec142a20 2024-03-25T19:19:50.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.716+0000 7f15daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:50.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.716+0000 7f15daffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:19:50.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.716+0000 7f15daffd640 10 monclient: dump: 2024-03-25T19:19:50.716 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T19:19:50.716 INFO:teuthology.orchestra.run.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:50.716 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:50.716 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:50.716 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:50.716 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 3 2024-03-25T19:19:50.717 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:50.717 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:50.717 INFO:teuthology.orchestra.run.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:50.717 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T19:19:50.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.716+0000 7f15daffd640 1 -- 172.21.15.83:0/1262036209 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f15e0003d40 con 0x7f15ec142a20 2024-03-25T19:19:50.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.716+0000 7f15daffd640 1 -- 172.21.15.83:0/1262036209 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7f15e0005690 con 0x7f15ec142a20 2024-03-25T19:19:50.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.716+0000 7f15daffd640 10 client.4167.objecter ms_dispatch 0x7f15ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:19:50.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.716+0000 7f15daffd640 3 client.4167.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:19:50.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.716+0000 7f15daffd640 3 client.4167.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:19:50.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.716+0000 7f15daffd640 20 client.4167.objecter dump_active .. 0 homeless 2024-03-25T19:19:50.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.716+0000 7f15daffd640 1 -- 172.21.15.83:0/1262036209 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f15e0005ab0 con 0x7f15ec142a20 2024-03-25T19:19:50.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.716+0000 7f15daffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:50.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.716+0000 7f15daffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:19:50.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.716+0000 7f15e8d99640 4 set_mon_vals no callback set 2024-03-25T19:19:50.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.846+0000 7f15d8ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:19:50.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.846+0000 7f15d8ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:19:50.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.846+0000 7f15d8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:19:50.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.846+0000 7f15d8ff9640 1 -- 172.21.15.83:0/1262036209 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f15d0005330 con 0x7f15ec142a20 2024-03-25T19:19:50.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.928+0000 7f15daffd640 1 -- 172.21.15.83:0/1262036209 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 3856139200 0 0) 0x7f15e0004f00 con 0x7f15ec142a20 2024-03-25T19:19:50.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.978+0000 7f15daffd640 1 -- 172.21.15.83:0/1262036209 <== mon.0 v1:172.21.15.83:6789/0 11 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7f15e0014720 con 0x7f15ec142a20 2024-03-25T19:19:50.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.978+0000 7f15daffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:19:50.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.978+0000 7f15daffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:19:50.979 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T19:19:50.979 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:19:50.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.979+0000 7f15d8ff9640 10 librados: watch_flush enter 2024-03-25T19:19:50.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.979+0000 7f15d8ff9640 10 librados: watch_flush exit 2024-03-25T19:19:50.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.979+0000 7f15d8ff9640 20 client.4167.objecter shutdown clearing up homeless session... 2024-03-25T19:19:50.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.979+0000 7f15d8ff9640 10 client.4167.objecter successfully canceled tick 2024-03-25T19:19:50.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.979+0000 7f15d8ff9640 10 monclient: shutdown 2024-03-25T19:19:50.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.979+0000 7f15d8ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:19:50.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.979+0000 7f15d8ff9640 1 -- 172.21.15.83:0/1262036209 >> v1:172.21.15.83:6789/0 conn(0x7f15ec142a20 legacy=0x7f15ec11fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:50.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.979+0000 7f15d8ff9640 1 -- 172.21.15.83:0/1262036209 shutdown_connections 2024-03-25T19:19:50.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.979+0000 7f15d8ff9640 1 -- 172.21.15.83:0/1262036209 >> 172.21.15.83:0/1262036209 conn(0x7f15ec0668e0 msgr2=0x7f15ec13f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:19:50.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.979+0000 7f15d8ff9640 1 -- 172.21.15.83:0/1262036209 shutdown_connections 2024-03-25T19:19:50.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.980+0000 7f15d8ff9640 1 -- 172.21.15.83:0/1262036209 wait complete. 2024-03-25T19:19:50.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T19:19:50.980+0000 7f15d8ff9640 1 librados: shutdown 2024-03-25T19:19:50.994 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-25T19:19:50.994 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-25T19:19:50.998 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-25T19:19:50.999+0000 7f54aa63c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T19:19:51.090 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-25T19:19:51.090+0000 7f54aa63c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T19:19:51.297 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-25T19:19:51.298+0000 7f54aa63c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T19:19:51.800 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:19:51.942 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.942+0000 7fabfc40f640 10 monclient: get_monmap_and_config 2024-03-25T19:19:51.943 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.942+0000 7fabfc40f640 10 monclient: build_initial_monmap 2024-03-25T19:19:51.943 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.943+0000 7fabfc40f640 10 monclient: monmap: 2024-03-25T19:19:51.943 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:19:51.943 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:51.943 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:19:51.943908+0000 2024-03-25T19:19:51.943 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:19:51.943908+0000 2024-03-25T19:19:51.943 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:19:51.943 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:19:51.943 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:19:51.943 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:19:51.943 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:19:51.943 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:19:51.944 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.943+0000 7fabfc40f640 1 Processor -- start 2024-03-25T19:19:51.945 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.943+0000 7fabfc40f640 1 -- start start 2024-03-25T19:19:51.945 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.943+0000 7fabfc40f640 10 monclient: init 2024-03-25T19:19:51.945 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.945+0000 7fabfc40f640 10 monclient: _reopen_session rank -1 2024-03-25T19:19:51.945 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.945+0000 7fabfc40f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:19:51.945 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.945+0000 7fabfc40f640 10 monclient(hunting): picked mon.noname-c con 0x7fabf40a98f0 addr v1:172.21.15.103:6789/0 2024-03-25T19:19:51.945 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.945+0000 7fabfc40f640 10 monclient(hunting): picked mon.noname-a con 0x7fabf413f710 addr v1:172.21.15.83:6789/0 2024-03-25T19:19:51.946 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.945+0000 7fabfc40f640 10 monclient(hunting): picked mon.noname-b con 0x7fabf41430c0 addr v1:172.21.15.83:6790/0 2024-03-25T19:19:51.946 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.945+0000 7fabfc40f640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabf4142520 con 0x7fabf413f710 2024-03-25T19:19:51.946 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.945+0000 7fabfc40f640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabf40b28a0 con 0x7fabf40a98f0 2024-03-25T19:19:51.946 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.945+0000 7fabfc40f640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabf40b2a30 con 0x7fabf41430c0 2024-03-25T19:19:51.946 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.945+0000 7fabfc40f640 10 monclient(hunting): _renew_subs 2024-03-25T19:19:51.946 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.945+0000 7fabfc40f640 10 monclient(hunting): authenticate will time out at 6049.217399s 2024-03-25T19:19:51.946 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.946+0000 7fabfa9ac640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fabf41430c0 0x7fabf414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:38984/0 (socket says 172.21.15.83:38984) 2024-03-25T19:19:51.947 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.946+0000 7fabfa9ac640 1 -- 172.21.15.83:0/3430702997 learned_addr learned my addr 172.21.15.83:0/3430702997 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:19:51.950 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.949+0000 7fabf91a9640 1 -- 172.21.15.83:0/3430702997 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1996068485 0 0) 0x7fabf40b2a30 con 0x7fabf41430c0 2024-03-25T19:19:51.950 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.949+0000 7fabf91a9640 10 monclient(hunting): _negotiate 2024-03-25T19:19:51.950 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.950+0000 7fabf91a9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:51.950 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.950+0000 7fabf91a9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:51.950 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.950+0000 7fabf91a9640 10 monclient(hunting): my global_id is 4184 2024-03-25T19:19:51.951 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.950+0000 7fabf91a9640 1 -- 172.21.15.83:0/3430702997 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabe0003520 con 0x7fabf41430c0 2024-03-25T19:19:51.951 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.950+0000 7fabf91a9640 1 -- 172.21.15.83:0/3430702997 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1237280458 0 0) 0x7fabf40b28a0 con 0x7fabf40a98f0 2024-03-25T19:19:51.951 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.950+0000 7fabf91a9640 10 monclient(hunting): _negotiate 2024-03-25T19:19:51.951 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.950+0000 7fabf91a9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:51.951 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.950+0000 7fabf91a9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:51.951 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.950+0000 7fabf91a9640 10 monclient(hunting): my global_id is 4177 2024-03-25T19:19:51.951 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.950+0000 7fabf91a9640 1 -- 172.21.15.83:0/3430702997 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabf40b2a30 con 0x7fabf40a98f0 2024-03-25T19:19:51.956 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.955+0000 7fabf91a9640 1 -- 172.21.15.83:0/3430702997 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1896576354 0 0) 0x7fabe0003520 con 0x7fabf41430c0 2024-03-25T19:19:51.956 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.955+0000 7fabf91a9640 1 -- 172.21.15.83:0/3430702997 >> v1:172.21.15.103:6789/0 conn(0x7fabf40a98f0 legacy=0x7fabf40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:51.956 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.955+0000 7fabf91a9640 1 -- 172.21.15.83:0/3430702997 >> v1:172.21.15.83:6789/0 conn(0x7fabf413f710 legacy=0x7fabf41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:51.956 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.955+0000 7fabf91a9640 10 monclient: _finish_hunting 0 2024-03-25T19:19:51.956 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.955+0000 7fabf91a9640 1 monclient: found mon.noname-b 2024-03-25T19:19:51.956 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.955+0000 7fabf91a9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:19:51.956 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.955+0000 7fabf91a9640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.83:6790/0 2024-03-25T19:19:51.956 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.955+0000 7fabf91a9640 1 -- 172.21.15.83:0/3430702997 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf40b44c0 con 0x7fabf41430c0 2024-03-25T19:19:51.957 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.955+0000 7fabf91a9640 10 monclient: _finish_auth 0 2024-03-25T19:19:51.957 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.955+0000 7fabf91a9640 10 monclient: _check_auth_tickets 2024-03-25T19:19:51.957 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.956+0000 7fabf91a9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:19:51.957 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.956+0000 7fabf91a9640 1 -- 172.21.15.83:0/3430702997 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fabec001a10 con 0x7fabf41430c0 2024-03-25T19:19:51.957 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.956+0000 7fabfc40f640 5 monclient: authenticate success, global_id 4184 2024-03-25T19:19:51.957 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.956+0000 7fabfc40f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:19:51.957 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.956+0000 7fabf91a9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:51.957 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.956+0000 7fabf91a9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:19:51.957 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.956+0000 7fabf91a9640 10 monclient: dump: 2024-03-25T19:19:51.957 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:19:51.957 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:51.957 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:51.957 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:51.957 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:51.957 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:19:51.958 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:51.958 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:51.958 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:51.958 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:19:51.958 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.956+0000 7fabfc40f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:19:51.959 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.958+0000 7fabf91a9640 1 -- 172.21.15.83:0/3430702997 <== mon.2 v1:172.21.15.83:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fabec001c30 con 0x7fabf41430c0 2024-03-25T19:19:51.959 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.958+0000 7fabf91a9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:19:51.959 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.958+0000 7fabf91a9640 1 -- 172.21.15.83:0/3430702997 <== mon.2 v1:172.21.15.83:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fabec001fc0 con 0x7fabf41430c0 2024-03-25T19:19:51.960 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.958+0000 7fabfc40f640 10 monclient: get_monmap_and_config success 2024-03-25T19:19:51.960 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.958+0000 7fabfc40f640 4 set_mon_vals no callback set 2024-03-25T19:19:51.960 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.959+0000 7fabf91a9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:51.960 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.959+0000 7fabf91a9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:19:51.960 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.959+0000 7fabf91a9640 10 monclient: dump: 2024-03-25T19:19:51.960 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:19:51.960 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:51.960 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:51.960 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:51.960 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:51.960 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:19:51.960 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:51.960 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:51.960 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:51.960 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:19:51.961 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.959+0000 7fabfc40f640 10 monclient: shutdown 2024-03-25T19:19:51.961 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.960+0000 7fabfc40f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:19:51.961 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.960+0000 7fabfc40f640 1 -- 172.21.15.83:0/3430702997 >> v1:172.21.15.83:6790/0 conn(0x7fabf41430c0 legacy=0x7fabf414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:51.962 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.960+0000 7fabfc40f640 1 -- 172.21.15.83:0/3430702997 shutdown_connections 2024-03-25T19:19:51.962 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.960+0000 7fabfc40f640 1 -- 172.21.15.83:0/3430702997 >> 172.21.15.83:0/3430702997 conn(0x7fabf40b7ea0 msgr2=0x7fabf40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:19:51.962 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.961+0000 7fabfc40f640 1 -- 172.21.15.83:0/3430702997 shutdown_connections 2024-03-25T19:19:51.962 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.961+0000 7fabfc40f640 1 -- 172.21.15.83:0/3430702997 wait complete. 2024-03-25T19:19:51.962 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.962+0000 7fabfc40f640 10 monclient: build_initial_monmap 2024-03-25T19:19:51.962 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.962+0000 7fabfc40f640 10 monclient: monmap: 2024-03-25T19:19:51.962 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:19:51.962 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:19:51.962 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:19:51.962 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:19:51.962 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:19:51.962 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:19:51.962 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:19:51.962 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:19:51.963 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:19:51.963 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:19:51.965 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.962+0000 7fabfc40f640 1 librados: starting msgr at 2024-03-25T19:19:51.965 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.963+0000 7fabfc40f640 1 librados: starting objecter 2024-03-25T19:19:51.965 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.963+0000 7fabfc40f640 1 Processor -- start 2024-03-25T19:19:51.965 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.963+0000 7fabfc40f640 1 -- start start 2024-03-25T19:19:51.965 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.963+0000 7fabfc40f640 1 librados: setting wanted keys 2024-03-25T19:19:51.965 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.963+0000 7fabfc40f640 1 librados: calling monclient init 2024-03-25T19:19:51.965 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.963+0000 7fabfc40f640 10 monclient: init 2024-03-25T19:19:51.965 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.964+0000 7fabfc40f640 10 monclient: _reopen_session rank -1 2024-03-25T19:19:51.965 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.964+0000 7fabfc40f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:19:51.965 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.964+0000 7fabfc40f640 10 monclient(hunting): picked mon.noname-c con 0x7fabf40a98f0 addr v1:172.21.15.83:6790/0 2024-03-25T19:19:51.965 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.964+0000 7fabfc40f640 10 monclient(hunting): picked mon.noname-a con 0x7fabf413f710 addr v1:172.21.15.83:6789/0 2024-03-25T19:19:51.965 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.965+0000 7fabfc40f640 10 monclient(hunting): picked mon.noname-b con 0x7fabf41430c0 addr v1:172.21.15.103:6789/0 2024-03-25T19:19:51.965 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.965+0000 7fabfc40f640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabf4138d70 con 0x7fabf413f710 2024-03-25T19:19:51.965 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.965+0000 7fabfc40f640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabf41dc070 con 0x7fabf41430c0 2024-03-25T19:19:51.966 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.965+0000 7fabfc40f640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabf4138fe0 con 0x7fabf40a98f0 2024-03-25T19:19:51.966 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.965+0000 7fabfc40f640 10 monclient(hunting): _renew_subs 2024-03-25T19:19:51.966 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.965+0000 7fabfc40f640 10 monclient(hunting): authenticate will time out at 6049.237166s 2024-03-25T19:19:51.967 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.965+0000 7fabfa9ac640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fabf41430c0 0x7fabf41db960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:49308/0 (socket says 172.21.15.83:49308) 2024-03-25T19:19:51.967 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.965+0000 7fabfa9ac640 1 -- 172.21.15.83:0/3919536912 learned_addr learned my addr 172.21.15.83:0/3919536912 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:19:51.967 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.966+0000 7fabeaffd640 10 client.?.objecter ms_handle_connect 0x7fabf41430c0 2024-03-25T19:19:51.967 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.966+0000 7fabeaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:51.968 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.968+0000 7fabeaffd640 1 -- 172.21.15.83:0/3919536912 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1779844633 0 0) 0x7fabf41dc070 con 0x7fabf41430c0 2024-03-25T19:19:51.968 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.968+0000 7fabeaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:19:51.968 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.968+0000 7fabeaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:51.968 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.968+0000 7fabeaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:51.968 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.968+0000 7fabeaffd640 10 monclient(hunting): my global_id is 4180 2024-03-25T19:19:51.968 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.968+0000 7fabeaffd640 1 -- 172.21.15.83:0/3919536912 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabd8004eb0 con 0x7fabf41430c0 2024-03-25T19:19:51.977 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.977+0000 7fabeaffd640 1 -- 172.21.15.83:0/3919536912 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 999607510 0 0) 0x7fabd8004eb0 con 0x7fabf41430c0 2024-03-25T19:19:51.977 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.977+0000 7fabeaffd640 1 -- 172.21.15.83:0/3919536912 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fabf41dc070 con 0x7fabf41430c0 2024-03-25T19:19:51.977 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.977+0000 7fabeaffd640 1 -- 172.21.15.83:0/3919536912 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fabec001f40 con 0x7fabf41430c0 2024-03-25T19:19:51.977 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.977+0000 7fabeaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:51.977 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.977+0000 7fabeaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:19:51.977 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.977+0000 7fabeaffd640 10 monclient(hunting): dump: 2024-03-25T19:19:51.977 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:19:51.977 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:51.977 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:51.978 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:51.978 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:51.978 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:19:51.978 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:51.978 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:51.978 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:51.978 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:19:51.978 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.977+0000 7fabeaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:19:51.986 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.984+0000 7fabeaffd640 1 -- 172.21.15.83:0/3919536912 <== mon.1 v1:172.21.15.103:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 390397077 0 0) 0x7fabf41dc070 con 0x7fabf41430c0 2024-03-25T19:19:51.986 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.984+0000 7fabeaffd640 1 -- 172.21.15.83:0/3919536912 >> v1:172.21.15.83:6790/0 conn(0x7fabf40a98f0 legacy=0x7fabf4137da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:51.986 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.984+0000 7fabeaffd640 1 -- 172.21.15.83:0/3919536912 >> v1:172.21.15.83:6789/0 conn(0x7fabf413f710 legacy=0x7fabf41db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:51.986 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.984+0000 7fabeaffd640 10 monclient: _finish_hunting 0 2024-03-25T19:19:51.986 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.984+0000 7fabeaffd640 1 monclient: found mon.b 2024-03-25T19:19:51.986 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.984+0000 7fabeaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:19:51.986 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.984+0000 7fabeaffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:19:51.986 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.984+0000 7fabeaffd640 1 -- 172.21.15.83:0/3919536912 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf41392c0 con 0x7fabf41430c0 2024-03-25T19:19:51.986 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.984+0000 7fabeaffd640 10 monclient: _finish_auth 0 2024-03-25T19:19:51.987 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.984+0000 7fabeaffd640 10 monclient: _check_auth_tickets 2024-03-25T19:19:51.987 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.984+0000 7fabeaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:19:51.987 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.985+0000 7fabfc40f640 5 monclient: authenticate success, global_id 4180 2024-03-25T19:19:51.987 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.985+0000 7fabfc40f640 10 monclient: _renew_subs 2024-03-25T19:19:51.987 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.985+0000 7fabfc40f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:19:51.987 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.985+0000 7fabfc40f640 1 -- 172.21.15.83:0/3919536912 --> v1:172.21.15.103:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabf41395a0 con 0x7fabf41430c0 2024-03-25T19:19:51.987 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.985+0000 7fabfc40f640 10 client.4180.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:19:51.987 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.985+0000 7fabfc40f640 10 monclient: _renew_subs 2024-03-25T19:19:51.987 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.985+0000 7fabfc40f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:19:51.987 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.986+0000 7fabfc40f640 1 -- 172.21.15.83:0/3919536912 --> v1:172.21.15.103:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fabf41384b0 con 0x7fabf41430c0 2024-03-25T19:19:51.987 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.986+0000 7fabfc40f640 1 librados: init done 2024-03-25T19:19:51.988 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.987+0000 7fabfc40f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:51.989 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.987+0000 7fabfc40f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:51.989 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.987+0000 7fabfc40f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:19:51.989 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.987+0000 7fabeaffd640 1 -- 172.21.15.83:0/3919536912 <== mon.1 v1:172.21.15.103:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fabec004e10 con 0x7fabf41430c0 2024-03-25T19:19:51.989 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.987+0000 7fabfc40f640 1 -- 172.21.15.83:0/3919536912 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabf40627b0 con 0x7fabf41430c0 2024-03-25T19:19:51.989 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.987+0000 7fabeaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:19:51.989 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.987+0000 7fabeaffd640 1 -- 172.21.15.83:0/3919536912 <== mon.1 v1:172.21.15.103:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fabec005260 con 0x7fabf41430c0 2024-03-25T19:19:51.989 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.987+0000 7fabeaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:51.989 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.987+0000 7fabeaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:19:51.989 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.987+0000 7fabeaffd640 10 monclient: dump: 2024-03-25T19:19:51.989 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:19:51.989 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:51.989 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:51.989 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:51.990 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:51.990 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:19:51.990 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:51.990 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:51.990 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:51.990 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:19:51.990 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:51.987+0000 7fabf89a8640 4 set_mon_vals no callback set 2024-03-25T19:19:52.217 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:52.217+0000 7fabeaffd640 1 -- 172.21.15.83:0/3919536912 <== mon.1 v1:172.21.15.103:6789/0 7 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 3856139200 0 0) 0x7fabec0053c0 con 0x7fabf41430c0 2024-03-25T19:19:52.223 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:52.223+0000 7fabeaffd640 1 -- 172.21.15.83:0/3919536912 <== mon.1 v1:172.21.15.103:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7fabec015c20 con 0x7fabf41430c0 2024-03-25T19:19:52.223 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:52.223+0000 7fabeaffd640 10 client.4180.objecter ms_dispatch 0x7fabf4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:19:52.223 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:52.223+0000 7fabeaffd640 3 client.4180.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:19:52.223 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:52.223+0000 7fabeaffd640 3 client.4180.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:19:52.223 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:52.223+0000 7fabeaffd640 20 client.4180.objecter dump_active .. 0 homeless 2024-03-25T19:19:52.223 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:52.223+0000 7fabeaffd640 1 -- 172.21.15.83:0/3919536912 <== mon.1 v1:172.21.15.103:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fabec0150c0 con 0x7fabf41430c0 2024-03-25T19:19:52.223 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:52.223+0000 7fabeaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:52.224 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:52.223+0000 7fabeaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:19:52.348 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:52.348+0000 7fabfc40f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:19:52.348 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:52.348+0000 7fabfc40f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:19:52.348 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:52.348+0000 7fabfc40f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:19:52.348 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:52.348+0000 7fabfc40f640 1 -- 172.21.15.83:0/3919536912 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fabf413df70 con 0x7fabf41430c0 2024-03-25T19:19:52.490 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-03-25T19:19:52.491+0000 7f02429fc640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-25T19:19:52.538 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:52.539+0000 7fabeaffd640 1 -- 172.21.15.83:0/3919536912 <== mon.1 v1:172.21.15.103:6789/0 10 ==== mgrmap(e 3) v1 ==== 89591+0+0 (unknown 3826116006 0 0) 0x7fabec057dc0 con 0x7fabf41430c0 2024-03-25T19:19:52.539 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:52.539+0000 7fabfa1ab640 10 monclient: get_auth_request con 0x7fabd806e0e0 auth_method 0 2024-03-25T19:19:52.539 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:52.540+0000 7fabeaffd640 10 client.4180.objecter ms_handle_connect 0x7fabd806e0e0 2024-03-25T19:19:52.594 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-03-25T19:19:52.595+0000 7f02429fc640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-25T19:19:52.798 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:52.798+0000 7fabeaffd640 1 -- 172.21.15.83:0/3919536912 <== mon.1 v1:172.21.15.103:6789/0 11 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7fabec00a240 con 0x7fabf41430c0 2024-03-25T19:19:52.798 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:52.798+0000 7fabeaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:19:52.798 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:52.798+0000 7fabeaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:19:52.798 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:19:52.799 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:19:52.800 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:52.799+0000 7fabfc40f640 10 librados: watch_flush enter 2024-03-25T19:19:52.800 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:52.799+0000 7fabfc40f640 10 librados: watch_flush exit 2024-03-25T19:19:52.800 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:52.800+0000 7fabfc40f640 20 client.4180.objecter shutdown clearing up homeless session... 2024-03-25T19:19:52.800 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:52.800+0000 7fabfc40f640 10 client.4180.objecter successfully canceled tick 2024-03-25T19:19:52.800 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:52.800+0000 7fabfc40f640 1 -- 172.21.15.83:0/3919536912 >> v1:172.21.15.103:6800/345072327 conn(0x7fabd806e0e0 legacy=0x7fabd80704d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:52.800 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:52.800+0000 7fabfc40f640 10 monclient: shutdown 2024-03-25T19:19:52.800 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:52.800+0000 7fabfc40f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:19:52.800 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:52.800+0000 7fabfc40f640 1 -- 172.21.15.83:0/3919536912 >> v1:172.21.15.103:6789/0 conn(0x7fabf41430c0 legacy=0x7fabf41db960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:52.800 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:52.800+0000 7fabfc40f640 1 -- 172.21.15.83:0/3919536912 shutdown_connections 2024-03-25T19:19:52.800 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:52.800+0000 7fabfc40f640 1 -- 172.21.15.83:0/3919536912 >> 172.21.15.83:0/3919536912 conn(0x7fabf40b7ea0 msgr2=0x7fabf413d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:19:52.800 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:52.800+0000 7fabfc40f640 1 -- 172.21.15.83:0/3919536912 shutdown_connections 2024-03-25T19:19:52.800 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:52.800+0000 7fabfc40f640 1 -- 172.21.15.83:0/3919536912 wait complete. 2024-03-25T19:19:52.801 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:52.800+0000 7fabfc40f640 1 librados: shutdown 2024-03-25T19:19:52.819 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:19:57.394 INFO:tasks.ceph.osd.5.smithi103.stderr:==00:00:00:06.887 35633== Warning: unimplemented fcntl command: 1038 2024-03-25T19:19:57.473 INFO:tasks.ceph.osd.5.smithi103.stderr:==00:00:00:06.966 35633== Warning: unimplemented fcntl command: 1038 2024-03-25T19:19:57.640 INFO:tasks.ceph.osd.7.smithi103.stderr:==00:00:00:07.126 35635== Warning: unimplemented fcntl command: 1038 2024-03-25T19:19:57.641 INFO:tasks.ceph.osd.6.smithi103.stderr:==00:00:00:07.126 35634== Warning: unimplemented fcntl command: 1038 2024-03-25T19:19:57.694 INFO:tasks.ceph.osd.0.smithi083.stderr:==00:00:00:07.190 35986== Warning: unimplemented fcntl command: 1038 2024-03-25T19:19:57.701 INFO:tasks.ceph.osd.7.smithi103.stderr:==00:00:00:07.188 35635== Warning: unimplemented fcntl command: 1038 2024-03-25T19:19:57.702 INFO:tasks.ceph.osd.6.smithi103.stderr:==00:00:00:07.186 35634== Warning: unimplemented fcntl command: 1038 2024-03-25T19:19:57.746 INFO:tasks.ceph.osd.4.smithi103.stderr:==00:00:00:07.256 35632== Warning: unimplemented fcntl command: 1038 2024-03-25T19:19:57.758 INFO:tasks.ceph.osd.0.smithi083.stderr:==00:00:00:07.254 35986== Warning: unimplemented fcntl command: 1038 2024-03-25T19:19:57.821 INFO:tasks.ceph.osd.4.smithi103.stderr:==00:00:00:07.330 35632== Warning: unimplemented fcntl command: 1038 2024-03-25T19:19:57.829 INFO:tasks.ceph.osd.1.smithi083.stderr:==00:00:00:07.325 35987== Warning: unimplemented fcntl command: 1038 2024-03-25T19:19:57.860 INFO:tasks.ceph.osd.3.smithi083.stderr:==00:00:00:07.344 35989== Warning: unimplemented fcntl command: 1038 2024-03-25T19:19:57.891 INFO:tasks.ceph.osd.1.smithi083.stderr:==00:00:00:07.386 35987== Warning: unimplemented fcntl command: 1038 2024-03-25T19:19:57.918 INFO:tasks.ceph.osd.3.smithi083.stderr:==00:00:00:07.403 35989== Warning: unimplemented fcntl command: 1038 2024-03-25T19:19:57.971 INFO:tasks.ceph.osd.2.smithi083.stderr:==00:00:00:07.463 35988== Warning: unimplemented fcntl command: 1038 2024-03-25T19:19:58.016 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-03-25T19:19:58.011+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:19:58.029 INFO:tasks.ceph.osd.2.smithi083.stderr:==00:00:00:07.521 35988== Warning: unimplemented fcntl command: 1038 2024-03-25T19:19:58.253 INFO:tasks.ceph.osd.7.smithi103.stderr:2024-03-25T19:19:58.248+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:19:58.285 INFO:tasks.ceph.osd.6.smithi103.stderr:2024-03-25T19:19:58.280+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:19:58.332 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T19:19:58.327+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:19:58.386 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-03-25T19:19:58.381+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:19:58.450 INFO:tasks.ceph.osd.1.smithi083.stderr:2024-03-25T19:19:58.445+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:19:58.478 INFO:tasks.ceph.osd.5.smithi103.stderr:==00:00:00:07.971 35633== Warning: unimplemented fcntl command: 1038 2024-03-25T19:19:58.489 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-03-25T19:19:58.481+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:19:58.597 INFO:tasks.ceph.osd.2.smithi083.stderr:2024-03-25T19:19:58.591+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:19:58.707 INFO:tasks.ceph.osd.7.smithi103.stderr:==00:00:00:08.193 35635== Warning: unimplemented fcntl command: 1038 2024-03-25T19:19:58.776 INFO:tasks.ceph.osd.6.smithi103.stderr:==00:00:00:08.261 35634== Warning: unimplemented fcntl command: 1038 2024-03-25T19:19:58.807 INFO:tasks.ceph.osd.0.smithi083.stderr:==00:00:00:08.303 35986== Warning: unimplemented fcntl command: 1038 2024-03-25T19:19:58.847 INFO:tasks.ceph.osd.5.smithi103.stderr:==00:00:00:08.339 35633== Warning: unimplemented fcntl command: 1038 2024-03-25T19:19:58.892 INFO:tasks.ceph.osd.4.smithi103.stderr:==00:00:00:08.401 35632== Warning: unimplemented fcntl command: 1038 2024-03-25T19:19:58.935 INFO:tasks.ceph.osd.1.smithi083.stderr:==00:00:00:08.430 35987== Warning: unimplemented fcntl command: 1038 2024-03-25T19:19:58.946 INFO:tasks.ceph.osd.3.smithi083.stderr:==00:00:00:08.431 35989== Warning: unimplemented fcntl command: 1038 2024-03-25T19:19:59.068 INFO:tasks.ceph.osd.2.smithi083.stderr:==00:00:00:08.560 35988== Warning: unimplemented fcntl command: 1038 2024-03-25T19:19:59.071 INFO:tasks.ceph.osd.7.smithi103.stderr:==00:00:00:08.557 35635== Warning: unimplemented fcntl command: 1038 2024-03-25T19:19:59.145 INFO:tasks.ceph.osd.6.smithi103.stderr:==00:00:00:08.630 35634== Warning: unimplemented fcntl command: 1038 2024-03-25T19:19:59.163 INFO:tasks.ceph.osd.5.smithi103.stderr:==00:00:00:08.655 35633== Warning: unimplemented fcntl command: 1038 2024-03-25T19:19:59.185 INFO:tasks.ceph.osd.0.smithi083.stderr:==00:00:00:08.682 35986== Warning: unimplemented fcntl command: 1038 2024-03-25T19:19:59.265 INFO:tasks.ceph.osd.4.smithi103.stderr:==00:00:00:08.775 35632== Warning: unimplemented fcntl command: 1038 2024-03-25T19:19:59.304 INFO:tasks.ceph.osd.1.smithi083.stderr:==00:00:00:08.800 35987== Warning: unimplemented fcntl command: 1038 2024-03-25T19:19:59.316 INFO:tasks.ceph.osd.3.smithi083.stderr:==00:00:00:08.801 35989== Warning: unimplemented fcntl command: 1038 2024-03-25T19:19:59.426 INFO:tasks.ceph.osd.7.smithi103.stderr:==00:00:00:08.912 35635== Warning: unimplemented fcntl command: 1038 2024-03-25T19:19:59.433 INFO:tasks.ceph.osd.2.smithi083.stderr:==00:00:00:08.925 35988== Warning: unimplemented fcntl command: 1038 2024-03-25T19:19:59.444 INFO:tasks.ceph.osd.5.smithi103.stderr:==00:00:00:08.937 35633== Warning: unimplemented fcntl command: 1038 2024-03-25T19:19:59.498 INFO:tasks.ceph.osd.0.smithi083.stderr:==00:00:00:08.994 35986== Warning: unimplemented fcntl command: 1038 2024-03-25T19:19:59.509 INFO:tasks.ceph.osd.6.smithi103.stderr:==00:00:00:08.993 35634== Warning: unimplemented fcntl command: 1038 2024-03-25T19:19:59.581 INFO:tasks.ceph.osd.4.smithi103.stderr:==00:00:00:09.090 35632== Warning: unimplemented fcntl command: 1038 2024-03-25T19:19:59.617 INFO:tasks.ceph.osd.1.smithi083.stderr:==00:00:00:09.113 35987== Warning: unimplemented fcntl command: 1038 2024-03-25T19:19:59.623 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:19:59.675 INFO:tasks.ceph.osd.3.smithi083.stderr:==00:00:00:09.160 35989== Warning: unimplemented fcntl command: 1038 2024-03-25T19:19:59.711 INFO:tasks.ceph.osd.5.smithi103.stderr:==00:00:00:09.204 35633== Warning: unimplemented fcntl command: 1038 2024-03-25T19:19:59.712 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.712+0000 7f3990f72640 10 monclient: get_monmap_and_config 2024-03-25T19:19:59.712 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.712+0000 7f3990f72640 10 monclient: build_initial_monmap 2024-03-25T19:19:59.712 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.712+0000 7f3990f72640 10 monclient: monmap: 2024-03-25T19:19:59.712 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:19:59.712 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:59.712 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:19:59.713732+0000 2024-03-25T19:19:59.712 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:19:59.713732+0000 2024-03-25T19:19:59.713 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:19:59.713 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:19:59.713 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:19:59.713 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:19:59.713 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:19:59.713 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:19:59.714 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.713+0000 7f3990f72640 1 Processor -- start 2024-03-25T19:19:59.714 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.713+0000 7f3990f72640 1 -- start start 2024-03-25T19:19:59.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.713+0000 7f3990f72640 10 monclient: init 2024-03-25T19:19:59.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.714+0000 7f3990f72640 10 monclient: _reopen_session rank -1 2024-03-25T19:19:59.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.714+0000 7f3990f72640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:19:59.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.714+0000 7f3990f72640 10 monclient(hunting): picked mon.noname-c con 0x7f398c06c380 addr v1:172.21.15.103:6789/0 2024-03-25T19:19:59.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.714+0000 7f3990f72640 10 monclient(hunting): picked mon.noname-a con 0x7f398c06d670 addr v1:172.21.15.83:6789/0 2024-03-25T19:19:59.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.714+0000 7f3990f72640 10 monclient(hunting): picked mon.noname-b con 0x7f398c0af8f0 addr v1:172.21.15.83:6790/0 2024-03-25T19:19:59.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.714+0000 7f3990f72640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f398c06c9a0 con 0x7f398c06d670 2024-03-25T19:19:59.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.714+0000 7f3990f72640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f398c0ada90 con 0x7f398c06c380 2024-03-25T19:19:59.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.714+0000 7f3990f72640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f398c0b0d10 con 0x7f398c0af8f0 2024-03-25T19:19:59.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.714+0000 7f3990f72640 10 monclient(hunting): _renew_subs 2024-03-25T19:19:59.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.714+0000 7f3990f72640 10 monclient(hunting): authenticate will time out at 6056.985972s 2024-03-25T19:19:59.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.714+0000 7f398ad9d640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f398c0af8f0 0x7f398c0aedf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:49152/0 (socket says 172.21.15.83:49152) 2024-03-25T19:19:59.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.714+0000 7f398ad9d640 1 -- 172.21.15.83:0/1503514422 learned_addr learned my addr 172.21.15.83:0/1503514422 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:19:59.717 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.717+0000 7f3989d9b640 1 -- 172.21.15.83:0/1503514422 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4077192838 0 0) 0x7f398c0b0d10 con 0x7f398c0af8f0 2024-03-25T19:19:59.717 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.717+0000 7f3989d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:19:59.717 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.717+0000 7f3989d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:59.717 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.717+0000 7f3989d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:59.717 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.717+0000 7f3989d9b640 10 monclient(hunting): my global_id is 4211 2024-03-25T19:19:59.717 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.717+0000 7f3989d9b640 1 -- 172.21.15.83:0/1503514422 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3978003520 con 0x7f398c0af8f0 2024-03-25T19:19:59.717 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.717+0000 7f3989d9b640 1 -- 172.21.15.83:0/1503514422 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4275471957 0 0) 0x7f398c06c9a0 con 0x7f398c06d670 2024-03-25T19:19:59.717 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.717+0000 7f3989d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:19:59.717 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.717+0000 7f3989d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:59.717 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.717+0000 7f3989d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:59.717 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.717+0000 7f3989d9b640 10 monclient(hunting): my global_id is 4200 2024-03-25T19:19:59.717 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.717+0000 7f3989d9b640 1 -- 172.21.15.83:0/1503514422 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f398c0b0d10 con 0x7f398c06d670 2024-03-25T19:19:59.718 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.717+0000 7f3989d9b640 1 -- 172.21.15.83:0/1503514422 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1602953857 0 0) 0x7f398c0ada90 con 0x7f398c06c380 2024-03-25T19:19:59.718 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.717+0000 7f3989d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:19:59.718 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.717+0000 7f3989d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:59.718 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.717+0000 7f3989d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:59.718 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.717+0000 7f3989d9b640 10 monclient(hunting): my global_id is 4207 2024-03-25T19:19:59.718 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.717+0000 7f3989d9b640 1 -- 172.21.15.83:0/1503514422 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f398c06c9a0 con 0x7f398c06c380 2024-03-25T19:19:59.720 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.720+0000 7f3989d9b640 1 -- 172.21.15.83:0/1503514422 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1110846465 0 0) 0x7f3978003520 con 0x7f398c0af8f0 2024-03-25T19:19:59.720 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.720+0000 7f3989d9b640 1 -- 172.21.15.83:0/1503514422 >> v1:172.21.15.103:6789/0 conn(0x7f398c06c380 legacy=0x7f398c0b0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:59.720 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.720+0000 7f3989d9b640 1 -- 172.21.15.83:0/1503514422 >> v1:172.21.15.83:6789/0 conn(0x7f398c06d670 legacy=0x7f398c0ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:59.720 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.720+0000 7f3989d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:19:59.720 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.720+0000 7f3989d9b640 1 monclient: found mon.noname-b 2024-03-25T19:19:59.720 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.720+0000 7f3989d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:19:59.720 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.720+0000 7f3989d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.83:6790/0 2024-03-25T19:19:59.720 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.720+0000 7f3989d9b640 1 -- 172.21.15.83:0/1503514422 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f398c0add00 con 0x7f398c0af8f0 2024-03-25T19:19:59.720 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.720+0000 7f3989d9b640 10 monclient: _finish_auth 0 2024-03-25T19:19:59.721 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.720+0000 7f3989d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:19:59.721 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.720+0000 7f3989d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:19:59.721 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.720+0000 7f3990f72640 5 monclient: authenticate success, global_id 4211 2024-03-25T19:19:59.721 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.720+0000 7f3990f72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:19:59.721 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.720+0000 7f3989d9b640 1 -- 172.21.15.83:0/1503514422 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f39840019d0 con 0x7f398c0af8f0 2024-03-25T19:19:59.721 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.720+0000 7f3989d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:59.721 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.720+0000 7f3989d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:19:59.721 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.720+0000 7f3989d9b640 10 monclient: dump: 2024-03-25T19:19:59.721 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:19:59.721 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:59.721 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:59.721 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:59.721 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:59.721 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:19:59.722 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:59.722 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:59.722 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:59.722 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:19:59.722 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.720+0000 7f3990f72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:19:59.722 INFO:tasks.ceph.osd.7.smithi103.stderr:==00:00:00:09.208 35635== Warning: unimplemented fcntl command: 1038 2024-03-25T19:19:59.725 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.722+0000 7f3989d9b640 1 -- 172.21.15.83:0/1503514422 <== mon.2 v1:172.21.15.83:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3984001bb0 con 0x7f398c0af8f0 2024-03-25T19:19:59.725 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.722+0000 7f3989d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:19:59.725 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.722+0000 7f3989d9b640 1 -- 172.21.15.83:0/1503514422 <== mon.2 v1:172.21.15.83:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f3984001f40 con 0x7f398c0af8f0 2024-03-25T19:19:59.725 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.722+0000 7f3989d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:59.725 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.722+0000 7f3989d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:19:59.725 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.722+0000 7f3989d9b640 10 monclient: dump: 2024-03-25T19:19:59.725 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:19:59.726 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:59.726 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:59.726 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:59.726 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:59.726 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:19:59.726 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:59.726 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:59.726 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:59.726 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:19:59.726 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.722+0000 7f3990f72640 10 monclient: get_monmap_and_config success 2024-03-25T19:19:59.726 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.722+0000 7f3990f72640 4 set_mon_vals no callback set 2024-03-25T19:19:59.726 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.722+0000 7f3990f72640 10 monclient: shutdown 2024-03-25T19:19:59.726 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.722+0000 7f3990f72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:19:59.727 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.722+0000 7f3990f72640 1 -- 172.21.15.83:0/1503514422 >> v1:172.21.15.83:6790/0 conn(0x7f398c0af8f0 legacy=0x7f398c0aedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:59.727 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.722+0000 7f3990f72640 1 -- 172.21.15.83:0/1503514422 shutdown_connections 2024-03-25T19:19:59.727 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.722+0000 7f3990f72640 1 -- 172.21.15.83:0/1503514422 >> 172.21.15.83:0/1503514422 conn(0x7f398c0b5c00 msgr2=0x7f398c0b5fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:19:59.727 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.723+0000 7f3990f72640 1 -- 172.21.15.83:0/1503514422 shutdown_connections 2024-03-25T19:19:59.727 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.723+0000 7f3990f72640 1 -- 172.21.15.83:0/1503514422 wait complete. 2024-03-25T19:19:59.727 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.723+0000 7f3990f72640 10 monclient: build_initial_monmap 2024-03-25T19:19:59.727 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.723+0000 7f3990f72640 10 monclient: monmap: 2024-03-25T19:19:59.727 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:19:59.727 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:19:59.727 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:19:59.727 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:19:59.727 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:19:59.727 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:19:59.727 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:19:59.728 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:19:59.728 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:19:59.728 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:19:59.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.724+0000 7f3990f72640 1 librados: starting msgr at 2024-03-25T19:19:59.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.724+0000 7f3990f72640 1 librados: starting objecter 2024-03-25T19:19:59.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.724+0000 7f3990f72640 1 Processor -- start 2024-03-25T19:19:59.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.724+0000 7f3990f72640 1 -- start start 2024-03-25T19:19:59.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.724+0000 7f3990f72640 1 librados: setting wanted keys 2024-03-25T19:19:59.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.724+0000 7f3990f72640 1 librados: calling monclient init 2024-03-25T19:19:59.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.724+0000 7f3990f72640 10 monclient: init 2024-03-25T19:19:59.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.724+0000 7f3990f72640 10 monclient: _reopen_session rank -1 2024-03-25T19:19:59.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.724+0000 7f3990f72640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:19:59.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.724+0000 7f3990f72640 10 monclient(hunting): picked mon.noname-c con 0x7f398c06c380 addr v1:172.21.15.83:6790/0 2024-03-25T19:19:59.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.724+0000 7f3990f72640 10 monclient(hunting): picked mon.noname-b con 0x7f398c06d670 addr v1:172.21.15.103:6789/0 2024-03-25T19:19:59.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.724+0000 7f3990f72640 10 monclient(hunting): picked mon.noname-a con 0x7f398c0af8f0 addr v1:172.21.15.83:6789/0 2024-03-25T19:19:59.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.724+0000 7f3990f72640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f398c145510 con 0x7f398c0af8f0 2024-03-25T19:19:59.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.724+0000 7f3990f72640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f398c1451c0 con 0x7f398c06d670 2024-03-25T19:19:59.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.724+0000 7f3990f72640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f398c1447e0 con 0x7f398c06c380 2024-03-25T19:19:59.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.724+0000 7f3990f72640 10 monclient(hunting): _renew_subs 2024-03-25T19:19:59.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.724+0000 7f3990f72640 10 monclient(hunting): authenticate will time out at 6056.996570s 2024-03-25T19:19:59.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.725+0000 7f398ad9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f398c0af8f0 0x7f398c1e2030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:50886/0 (socket says 172.21.15.83:50886) 2024-03-25T19:19:59.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.725+0000 7f398ad9d640 1 -- 172.21.15.83:0/3086621125 learned_addr learned my addr 172.21.15.83:0/3086621125 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:19:59.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.725+0000 7f3982ffd640 10 client.?.objecter ms_handle_connect 0x7f398c06c380 2024-03-25T19:19:59.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.725+0000 7f3982ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:59.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.725+0000 7f3982ffd640 10 client.?.objecter ms_handle_connect 0x7f398c0af8f0 2024-03-25T19:19:59.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.725+0000 7f3982ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:59.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.725+0000 7f3982ffd640 10 client.?.objecter ms_handle_connect 0x7f398c06d670 2024-03-25T19:19:59.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.725+0000 7f3982ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:19:59.730 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.727+0000 7f3982ffd640 1 -- 172.21.15.83:0/3086621125 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4207307447 0 0) 0x7f398c1447e0 con 0x7f398c06c380 2024-03-25T19:19:59.730 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.727+0000 7f3982ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:19:59.731 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.727+0000 7f3982ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:59.731 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.727+0000 7f3982ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:59.731 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.727+0000 7f3982ffd640 10 monclient(hunting): my global_id is 4214 2024-03-25T19:19:59.731 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.727+0000 7f3982ffd640 1 -- 172.21.15.83:0/3086621125 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f396c004eb0 con 0x7f398c06c380 2024-03-25T19:19:59.731 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.727+0000 7f3982ffd640 1 -- 172.21.15.83:0/3086621125 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3332113767 0 0) 0x7f398c145510 con 0x7f398c0af8f0 2024-03-25T19:19:59.731 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.727+0000 7f3982ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:19:59.731 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.727+0000 7f3982ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:59.731 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.727+0000 7f3982ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:59.731 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.727+0000 7f3982ffd640 10 monclient(hunting): my global_id is 4203 2024-03-25T19:19:59.731 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.727+0000 7f3982ffd640 1 -- 172.21.15.83:0/3086621125 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f398c1447e0 con 0x7f398c0af8f0 2024-03-25T19:19:59.731 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.727+0000 7f3982ffd640 1 -- 172.21.15.83:0/3086621125 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3436790965 0 0) 0x7f398c1451c0 con 0x7f398c06d670 2024-03-25T19:19:59.731 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.727+0000 7f3982ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:19:59.731 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.727+0000 7f3982ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:19:59.731 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.727+0000 7f3982ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:19:59.732 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.727+0000 7f3982ffd640 10 monclient(hunting): my global_id is 4210 2024-03-25T19:19:59.732 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.727+0000 7f3982ffd640 1 -- 172.21.15.83:0/3086621125 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f398c145510 con 0x7f398c06d670 2024-03-25T19:19:59.733 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.731+0000 7f3982ffd640 1 -- 172.21.15.83:0/3086621125 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1231639053 0 0) 0x7f398c1447e0 con 0x7f398c0af8f0 2024-03-25T19:19:59.733 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.731+0000 7f3982ffd640 1 -- 172.21.15.83:0/3086621125 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f398c1451c0 con 0x7f398c0af8f0 2024-03-25T19:19:59.733 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.732+0000 7f3982ffd640 1 -- 172.21.15.83:0/3086621125 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f3984001ec0 con 0x7f398c0af8f0 2024-03-25T19:19:59.733 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.732+0000 7f3982ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:59.733 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.732+0000 7f3982ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:19:59.733 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.732+0000 7f3982ffd640 10 monclient(hunting): dump: 2024-03-25T19:19:59.733 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:19:59.733 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:59.733 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:59.733 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:59.733 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:59.734 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:19:59.734 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:59.734 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:59.734 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:59.734 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:19:59.734 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.732+0000 7f3982ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:19:59.734 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.732+0000 7f3982ffd640 1 -- 172.21.15.83:0/3086621125 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2425032269 0 0) 0x7f396c004eb0 con 0x7f398c06c380 2024-03-25T19:19:59.734 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.732+0000 7f3982ffd640 1 -- 172.21.15.83:0/3086621125 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f398c1447e0 con 0x7f398c06c380 2024-03-25T19:19:59.734 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.732+0000 7f3982ffd640 1 -- 172.21.15.83:0/3086621125 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f397c0018f0 con 0x7f398c06c380 2024-03-25T19:19:59.734 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.732+0000 7f3982ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:59.734 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.732+0000 7f3982ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:19:59.734 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.732+0000 7f3982ffd640 10 monclient(hunting): dump: 2024-03-25T19:19:59.734 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:19:59.734 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:59.735 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:59.735 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:59.735 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:59.735 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:19:59.735 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:59.735 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:59.735 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:59.735 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:19:59.736 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.734+0000 7f3982ffd640 1 -- 172.21.15.83:0/3086621125 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2673259838 0 0) 0x7f398c145510 con 0x7f398c06d670 2024-03-25T19:19:59.736 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.734+0000 7f3982ffd640 1 -- 172.21.15.83:0/3086621125 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f396c004eb0 con 0x7f398c06d670 2024-03-25T19:19:59.736 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.734+0000 7f3982ffd640 1 -- 172.21.15.83:0/3086621125 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f39740018b0 con 0x7f398c06d670 2024-03-25T19:19:59.736 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.734+0000 7f3982ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:59.736 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.734+0000 7f3982ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:19:59.736 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.734+0000 7f3982ffd640 10 monclient(hunting): dump: 2024-03-25T19:19:59.737 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:19:59.737 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:59.737 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:59.737 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:59.737 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:59.737 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:19:59.737 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:59.737 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:59.737 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:59.737 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:19:59.737 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.734+0000 7f3982ffd640 1 -- 172.21.15.83:0/3086621125 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1655566079 0 0) 0x7f398c1451c0 con 0x7f398c0af8f0 2024-03-25T19:19:59.737 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.734+0000 7f3982ffd640 1 -- 172.21.15.83:0/3086621125 >> v1:172.21.15.83:6790/0 conn(0x7f398c06c380 legacy=0x7f398c146ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:59.737 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.734+0000 7f3982ffd640 1 -- 172.21.15.83:0/3086621125 >> v1:172.21.15.103:6789/0 conn(0x7f398c06d670 legacy=0x7f398c142f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:19:59.737 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.734+0000 7f3982ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:19:59.737 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.734+0000 7f3982ffd640 1 monclient: found mon.a 2024-03-25T19:19:59.737 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.734+0000 7f3982ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:19:59.737 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.734+0000 7f3982ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:19:59.738 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.734+0000 7f3982ffd640 1 -- 172.21.15.83:0/3086621125 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f398c144ac0 con 0x7f398c0af8f0 2024-03-25T19:19:59.738 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.734+0000 7f3982ffd640 10 monclient: _finish_auth 0 2024-03-25T19:19:59.738 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.734+0000 7f3982ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:19:59.738 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.734+0000 7f3982ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:19:59.738 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.734+0000 7f3990f72640 5 monclient: authenticate success, global_id 4203 2024-03-25T19:19:59.738 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.734+0000 7f3990f72640 10 monclient: _renew_subs 2024-03-25T19:19:59.738 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.734+0000 7f3990f72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:19:59.738 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.734+0000 7f3990f72640 1 -- 172.21.15.83:0/3086621125 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f398c144da0 con 0x7f398c0af8f0 2024-03-25T19:19:59.738 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.734+0000 7f3990f72640 10 client.4203.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:19:59.738 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.734+0000 7f3990f72640 10 monclient: _renew_subs 2024-03-25T19:19:59.738 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.734+0000 7f3990f72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:19:59.738 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.734+0000 7f3990f72640 1 -- 172.21.15.83:0/3086621125 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f398c143630 con 0x7f398c0af8f0 2024-03-25T19:19:59.738 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.734+0000 7f3990f72640 1 librados: init done 2024-03-25T19:19:59.738 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.735+0000 7f3990f72640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:59.739 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.735+0000 7f3990f72640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:59.739 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.735+0000 7f3990f72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:19:59.739 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.735+0000 7f3990f72640 1 -- 172.21.15.83:0/3086621125 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f398c0af480 con 0x7f398c0af8f0 2024-03-25T19:19:59.765 INFO:tasks.ceph.osd.0.smithi083.stderr:==00:00:00:09.261 35986== Warning: unimplemented fcntl command: 1038 2024-03-25T19:19:59.795 INFO:tasks.ceph.osd.2.smithi083.stderr:==00:00:00:09.287 35988== Warning: unimplemented fcntl command: 1038 2024-03-25T19:19:59.799 INFO:tasks.ceph.osd.6.smithi103.stderr:==00:00:00:09.284 35634== Warning: unimplemented fcntl command: 1038 2024-03-25T19:19:59.849 INFO:tasks.ceph.osd.4.smithi103.stderr:==00:00:00:09.358 35632== Warning: unimplemented fcntl command: 1038 2024-03-25T19:19:59.886 INFO:tasks.ceph.osd.1.smithi083.stderr:==00:00:00:09.381 35987== Warning: unimplemented fcntl command: 1038 2024-03-25T19:19:59.967 INFO:tasks.ceph.osd.3.smithi083.stderr:==00:00:00:09.451 35989== Warning: unimplemented fcntl command: 1038 2024-03-25T19:19:59.976 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.976+0000 7f3982ffd640 1 -- 172.21.15.83:0/3086621125 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3984002080 con 0x7f398c0af8f0 2024-03-25T19:19:59.976 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.976+0000 7f3982ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:19:59.976 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.976+0000 7f3982ffd640 1 -- 172.21.15.83:0/3086621125 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f3984004e10 con 0x7f398c0af8f0 2024-03-25T19:19:59.976 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.976+0000 7f3982ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:19:59.976 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.976+0000 7f3982ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:19:59.976 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.976+0000 7f3982ffd640 10 monclient: dump: 2024-03-25T19:19:59.976 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:19:59.976 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:19:59.976 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:59.976 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:19:59.976 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:19:59.976 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:19:59.976 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:19:59.976 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:19:59.977 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:19:59.977 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:19:59.977 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.976+0000 7f398959a640 4 set_mon_vals no callback set 2024-03-25T19:19:59.977 INFO:tasks.ceph.osd.5.smithi103.stderr:==00:00:00:09.469 35633== Warning: unimplemented fcntl command: 1038 2024-03-25T19:19:59.978 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.977+0000 7f3982ffd640 1 -- 172.21.15.83:0/3086621125 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7f398401ae60 con 0x7f398c0af8f0 2024-03-25T19:19:59.978 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.977+0000 7f398a59c640 10 monclient: get_auth_request con 0x7f396c06e140 auth_method 0 2024-03-25T19:19:59.978 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.978+0000 7f3982ffd640 10 client.4203.objecter ms_handle_connect 0x7f396c06e140 2024-03-25T19:19:59.985 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.985+0000 7f3982ffd640 1 -- 172.21.15.83:0/3086621125 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7f3984054b50 con 0x7f398c0af8f0 2024-03-25T19:19:59.986 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.985+0000 7f3982ffd640 10 client.4203.objecter ms_dispatch 0x7f398c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:19:59.986 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.985+0000 7f3982ffd640 3 client.4203.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:19:59.986 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.985+0000 7f3982ffd640 3 client.4203.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:19:59.986 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.986+0000 7f3982ffd640 20 client.4203.objecter dump_active .. 0 homeless 2024-03-25T19:19:59.986 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.986+0000 7f3982ffd640 1 -- 172.21.15.83:0/3086621125 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3984050490 con 0x7f398c0af8f0 2024-03-25T19:19:59.986 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.986+0000 7f3982ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:19:59.986 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:19:59.986+0000 7f3982ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:20:00.014 INFO:tasks.ceph.osd.7.smithi103.stderr:==00:00:00:09.500 35635== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:00.037 INFO:tasks.ceph.osd.0.smithi083.stderr:==00:00:00:09.533 35986== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:00.086 INFO:tasks.ceph.osd.2.smithi083.stderr:==00:00:00:09.578 35988== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:00.087 INFO:tasks.ceph.osd.6.smithi103.stderr:==00:00:00:09.572 35634== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:00.123 INFO:tasks.ceph.osd.4.smithi103.stderr:==00:00:00:09.632 35632== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:00.150 INFO:tasks.ceph.osd.1.smithi083.stderr:==00:00:00:09.645 35987== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:00.175 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:00.175+0000 7f3990f72640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:00.175 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:00.175+0000 7f3990f72640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:00.175 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:00.175+0000 7f3990f72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:20:00.175 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:00.175+0000 7f3990f72640 1 -- 172.21.15.83:0/3086621125 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f398c1456c0 con 0x7f398c0af8f0 2024-03-25T19:20:00.188 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:00.188+0000 7f3982ffd640 1 -- 172.21.15.83:0/3086621125 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7f398c1456c0 con 0x7f398c0af8f0 2024-03-25T19:20:00.188 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:00.188+0000 7f3982ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:00.188 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:00.188+0000 7f3982ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:20:00.188 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:20:00.189 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:20:00.191 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:00.190+0000 7f3990f72640 10 librados: watch_flush enter 2024-03-25T19:20:00.191 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:00.191+0000 7f3990f72640 10 librados: watch_flush exit 2024-03-25T19:20:00.191 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:00.191+0000 7f3990f72640 20 client.4203.objecter shutdown clearing up homeless session... 2024-03-25T19:20:00.191 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:00.191+0000 7f3990f72640 10 client.4203.objecter successfully canceled tick 2024-03-25T19:20:00.191 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:00.191+0000 7f3990f72640 1 -- 172.21.15.83:0/3086621125 >> v1:172.21.15.103:6800/345072327 conn(0x7f396c06e140 legacy=0x7f396c070600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:00.191 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:00.191+0000 7f3990f72640 10 monclient: shutdown 2024-03-25T19:20:00.191 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:00.191+0000 7f3990f72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:00.191 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:00.191+0000 7f3990f72640 1 -- 172.21.15.83:0/3086621125 >> v1:172.21.15.83:6789/0 conn(0x7f398c0af8f0 legacy=0x7f398c1e2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:00.191 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:00.191+0000 7f3990f72640 1 -- 172.21.15.83:0/3086621125 shutdown_connections 2024-03-25T19:20:00.192 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:00.191+0000 7f3990f72640 1 -- 172.21.15.83:0/3086621125 >> 172.21.15.83:0/3086621125 conn(0x7f398c0b5c00 msgr2=0x7f398c14e340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:00.192 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:00.191+0000 7f3990f72640 1 -- 172.21.15.83:0/3086621125 shutdown_connections 2024-03-25T19:20:00.192 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:00.191+0000 7f3990f72640 1 -- 172.21.15.83:0/3086621125 wait complete. 2024-03-25T19:20:00.192 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:00.191+0000 7f3990f72640 1 librados: shutdown 2024-03-25T19:20:00.200 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:20:00.245 INFO:tasks.ceph.osd.5.smithi103.stderr:==00:00:00:09.737 35633== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:00.254 INFO:tasks.ceph.osd.3.smithi083.stderr:==00:00:00:09.739 35989== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:00.301 INFO:tasks.ceph.osd.0.smithi083.stderr:==00:00:00:09.797 35986== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:00.305 INFO:tasks.ceph.osd.7.smithi103.stderr:==00:00:00:09.792 35635== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:00.375 INFO:tasks.ceph.osd.6.smithi103.stderr:==00:00:00:09.860 35634== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:00.390 INFO:tasks.ceph.osd.2.smithi083.stderr:==00:00:00:09.882 35988== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:00.395 INFO:tasks.ceph.osd.4.smithi103.stderr:==00:00:00:09.905 35632== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:00.420 INFO:tasks.ceph.osd.1.smithi083.stderr:==00:00:00:09.915 35987== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:00.509 INFO:tasks.ceph.osd.5.smithi103.stderr:==00:00:00:10.001 35633== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:00.542 INFO:tasks.ceph.osd.3.smithi083.stderr:==00:00:00:10.026 35989== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:00.577 INFO:tasks.ceph.osd.0.smithi083.stderr:==00:00:00:10.073 35986== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:00.596 INFO:tasks.ceph.osd.7.smithi103.stderr:==00:00:00:10.082 35635== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:00.640 INFO:tasks.ceph.osd.6.smithi103.stderr:==00:00:00:10.125 35634== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:00.662 INFO:tasks.ceph.osd.4.smithi103.stderr:==00:00:00:10.172 35632== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:00.684 INFO:tasks.ceph.osd.2.smithi083.stderr:==00:00:00:10.177 35988== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:00.692 INFO:tasks.ceph.osd.1.smithi083.stderr:==00:00:00:10.187 35987== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:00.778 INFO:tasks.ceph.osd.5.smithi103.stderr:==00:00:00:10.270 35633== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:00.829 INFO:tasks.ceph.osd.3.smithi083.stderr:==00:00:00:10.314 35989== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:00.843 INFO:tasks.ceph.osd.0.smithi083.stderr:==00:00:00:10.339 35986== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:00.884 INFO:tasks.ceph.osd.7.smithi103.stderr:==00:00:00:10.370 35635== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:00.933 INFO:tasks.ceph.osd.4.smithi103.stderr:==00:00:00:10.443 35632== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:00.942 INFO:tasks.ceph.osd.6.smithi103.stderr:==00:00:00:10.427 35634== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:00.956 INFO:tasks.ceph.osd.1.smithi083.stderr:==00:00:00:10.451 35987== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:00.980 INFO:tasks.ceph.osd.2.smithi083.stderr:==00:00:00:10.472 35988== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:01.108 INFO:tasks.ceph.osd.0.smithi083.stderr:==00:00:00:10.604 35986== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:01.132 INFO:tasks.ceph.osd.3.smithi083.stderr:==00:00:00:10.616 35989== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:01.171 INFO:tasks.ceph.osd.7.smithi103.stderr:==00:00:00:10.658 35635== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:01.198 INFO:tasks.ceph.osd.4.smithi103.stderr:==00:00:00:10.708 35632== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:01.220 INFO:tasks.ceph.osd.1.smithi083.stderr:==00:00:00:10.715 35987== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:01.231 INFO:tasks.ceph.osd.6.smithi103.stderr:==00:00:00:10.716 35634== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:01.262 INFO:tasks.ceph.osd.5.smithi103.stderr:==00:00:00:10.754 35633== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:01.267 INFO:tasks.ceph.osd.2.smithi083.stderr:==00:00:00:10.759 35988== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:01.314 INFO:tasks.ceph.osd.5.smithi103.stderr:==00:00:00:10.807 35633== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:01.437 INFO:tasks.ceph.osd.3.smithi083.stderr:==00:00:00:10.922 35989== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:01.561 INFO:tasks.ceph.osd.2.smithi083.stderr:==00:00:00:11.053 35988== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:01.602 INFO:tasks.ceph.osd.0.smithi083.stderr:==00:00:00:11.099 35986== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:01.655 INFO:tasks.ceph.osd.0.smithi083.stderr:==00:00:00:11.151 35986== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:01.695 INFO:tasks.ceph.osd.7.smithi103.stderr:==00:00:00:11.181 35635== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:01.697 INFO:tasks.ceph.osd.1.smithi083.stderr:==00:00:00:11.193 35987== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:01.721 INFO:tasks.ceph.osd.4.smithi103.stderr:==00:00:00:11.230 35632== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:01.749 INFO:tasks.ceph.osd.7.smithi103.stderr:==00:00:00:11.236 35635== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:01.751 INFO:tasks.ceph.osd.1.smithi083.stderr:==00:00:00:11.247 35987== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:01.817 INFO:tasks.ceph.osd.4.smithi103.stderr:==00:00:00:11.327 35632== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:01.833 INFO:tasks.ceph.osd.6.smithi103.stderr:==00:00:00:11.318 35634== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:01.888 INFO:tasks.ceph.osd.6.smithi103.stderr:==00:00:00:11.372 35634== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:01.956 INFO:tasks.ceph.osd.3.smithi083.stderr:==00:00:00:11.440 35989== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:02.011 INFO:tasks.ceph.osd.3.smithi083.stderr:==00:00:00:11.496 35989== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:02.114 INFO:tasks.ceph.osd.2.smithi083.stderr:==00:00:00:11.606 35988== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:02.170 INFO:tasks.ceph.osd.2.smithi083.stderr:==00:00:00:11.662 35988== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:06.712 INFO:tasks.ceph.osd.5.smithi103.stderr:==00:00:00:16.205 35633== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:07.006 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:20:07.095 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.095+0000 7f0d03954640 10 monclient: get_monmap_and_config 2024-03-25T19:20:07.095 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.095+0000 7f0d03954640 10 monclient: build_initial_monmap 2024-03-25T19:20:07.095 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.095+0000 7f0d03954640 10 monclient: monmap: 2024-03-25T19:20:07.095 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:20:07.095 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:20:07.095 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:20:07.096285+0000 2024-03-25T19:20:07.095 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:20:07.096285+0000 2024-03-25T19:20:07.095 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:07.095 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:20:07.095 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:20:07.095 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:20:07.095 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:20:07.095 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:20:07.097 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.095+0000 7f0d03954640 1 Processor -- start 2024-03-25T19:20:07.097 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.096+0000 7f0d03954640 1 -- start start 2024-03-25T19:20:07.097 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.096+0000 7f0d03954640 10 monclient: init 2024-03-25T19:20:07.097 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.096+0000 7f0d03954640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:07.097 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.096+0000 7f0d03954640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:20:07.097 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.096+0000 7f0d03954640 10 monclient(hunting): picked mon.noname-a con 0x7f0cfc0b4d60 addr v1:172.21.15.83:6789/0 2024-03-25T19:20:07.097 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.096+0000 7f0d03954640 10 monclient(hunting): picked mon.noname-c con 0x7f0cfc0b1730 addr v1:172.21.15.103:6789/0 2024-03-25T19:20:07.097 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.096+0000 7f0d03954640 10 monclient(hunting): picked mon.noname-b con 0x7f0cfc14be90 addr v1:172.21.15.83:6790/0 2024-03-25T19:20:07.097 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.096+0000 7f0d03954640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cfc0af940 con 0x7f0cfc0b4d60 2024-03-25T19:20:07.097 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.096+0000 7f0d03954640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cfc0b0a70 con 0x7f0cfc0b1730 2024-03-25T19:20:07.097 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.096+0000 7f0d03954640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cfc0b0d50 con 0x7f0cfc14be90 2024-03-25T19:20:07.098 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.096+0000 7f0d03954640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:07.098 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.096+0000 7f0d03954640 10 monclient(hunting): authenticate will time out at 6064.368605s 2024-03-25T19:20:07.098 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.097+0000 7f0d016f0640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f0cfc0b4d60 0x7f0cfc13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:54366/0 (socket says 172.21.15.83:54366) 2024-03-25T19:20:07.098 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.097+0000 7f0d016f0640 1 -- 172.21.15.83:0/439590956 learned_addr learned my addr 172.21.15.83:0/439590956 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:20:07.101 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.100+0000 7f0cf3fff640 1 -- 172.21.15.83:0/439590956 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3716801544 0 0) 0x7f0cfc0af940 con 0x7f0cfc0b4d60 2024-03-25T19:20:07.101 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.100+0000 7f0cf3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:20:07.101 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.100+0000 7f0cf3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:07.101 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.100+0000 7f0cf3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:07.101 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.100+0000 7f0cf3fff640 10 monclient(hunting): my global_id is 4206 2024-03-25T19:20:07.101 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.100+0000 7f0cf3fff640 1 -- 172.21.15.83:0/439590956 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ce8003520 con 0x7f0cfc0b4d60 2024-03-25T19:20:07.101 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.100+0000 7f0cf3fff640 1 -- 172.21.15.83:0/439590956 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1599321138 0 0) 0x7f0cfc0b0a70 con 0x7f0cfc0b1730 2024-03-25T19:20:07.101 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.100+0000 7f0cf3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:20:07.101 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.100+0000 7f0cf3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:07.101 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.100+0000 7f0cf3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:07.101 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.100+0000 7f0cf3fff640 10 monclient(hunting): my global_id is 4213 2024-03-25T19:20:07.101 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.100+0000 7f0cf3fff640 1 -- 172.21.15.83:0/439590956 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cfc0af940 con 0x7f0cfc0b1730 2024-03-25T19:20:07.102 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.101+0000 7f0cf3fff640 1 -- 172.21.15.83:0/439590956 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4061895019 0 0) 0x7f0cfc0b0d50 con 0x7f0cfc14be90 2024-03-25T19:20:07.102 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.101+0000 7f0cf3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:20:07.102 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.101+0000 7f0cf3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:07.102 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.101+0000 7f0cf3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:07.102 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.101+0000 7f0cf3fff640 10 monclient(hunting): my global_id is 4217 2024-03-25T19:20:07.102 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.101+0000 7f0cf3fff640 1 -- 172.21.15.83:0/439590956 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cfc0b0a70 con 0x7f0cfc14be90 2024-03-25T19:20:07.104 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.104+0000 7f0cf3fff640 1 -- 172.21.15.83:0/439590956 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2843732283 0 0) 0x7f0ce8003520 con 0x7f0cfc0b4d60 2024-03-25T19:20:07.104 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.104+0000 7f0cf3fff640 1 -- 172.21.15.83:0/439590956 >> v1:172.21.15.83:6790/0 conn(0x7f0cfc14be90 legacy=0x7f0cfc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:07.104 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.104+0000 7f0cf3fff640 1 -- 172.21.15.83:0/439590956 >> v1:172.21.15.103:6789/0 conn(0x7f0cfc0b1730 legacy=0x7f0cfc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:07.104 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.104+0000 7f0cf3fff640 10 monclient: _finish_hunting 0 2024-03-25T19:20:07.104 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.104+0000 7f0cf3fff640 1 monclient: found mon.noname-a 2024-03-25T19:20:07.104 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.104+0000 7f0cf3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:07.104 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.104+0000 7f0cf3fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.83:6789/0 2024-03-25T19:20:07.104 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.104+0000 7f0cf3fff640 1 -- 172.21.15.83:0/439590956 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cfc0b1030 con 0x7f0cfc0b4d60 2024-03-25T19:20:07.105 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.104+0000 7f0cf3fff640 10 monclient: _finish_auth 0 2024-03-25T19:20:07.105 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.104+0000 7f0cf3fff640 10 monclient: _check_auth_tickets 2024-03-25T19:20:07.105 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.104+0000 7f0cf3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:07.105 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.104+0000 7f0d03954640 5 monclient: authenticate success, global_id 4206 2024-03-25T19:20:07.105 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.104+0000 7f0d03954640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:20:07.105 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.104+0000 7f0cf3fff640 1 -- 172.21.15.83:0/439590956 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f0cf8001690 con 0x7f0cfc0b4d60 2024-03-25T19:20:07.105 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.104+0000 7f0cf3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:07.105 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.104+0000 7f0cf3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:20:07.105 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.104+0000 7f0cf3fff640 10 monclient: dump: 2024-03-25T19:20:07.105 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:20:07.105 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:20:07.105 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:07.105 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:07.105 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:07.105 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:20:07.105 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:20:07.106 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:20:07.106 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:20:07.106 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:20:07.106 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.104+0000 7f0d03954640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:20:07.107 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.106+0000 7f0cf3fff640 1 -- 172.21.15.83:0/439590956 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0cf8001870 con 0x7f0cfc0b4d60 2024-03-25T19:20:07.107 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.106+0000 7f0cf3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:07.108 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.106+0000 7f0cf3fff640 1 -- 172.21.15.83:0/439590956 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f0cf8001690 con 0x7f0cfc0b4d60 2024-03-25T19:20:07.108 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.106+0000 7f0cf3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:07.108 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.106+0000 7f0cf3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:20:07.108 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.106+0000 7f0cf3fff640 10 monclient: dump: 2024-03-25T19:20:07.108 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:20:07.108 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:20:07.108 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:07.108 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:07.108 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:07.108 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:20:07.108 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:20:07.108 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:20:07.108 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:20:07.109 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:20:07.109 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.106+0000 7f0d03954640 10 monclient: get_monmap_and_config success 2024-03-25T19:20:07.109 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.106+0000 7f0d03954640 4 set_mon_vals no callback set 2024-03-25T19:20:07.109 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.107+0000 7f0d03954640 10 monclient: shutdown 2024-03-25T19:20:07.109 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.107+0000 7f0d03954640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:07.109 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.107+0000 7f0d03954640 1 -- 172.21.15.83:0/439590956 >> v1:172.21.15.83:6789/0 conn(0x7f0cfc0b4d60 legacy=0x7f0cfc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:07.109 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.107+0000 7f0d03954640 1 -- 172.21.15.83:0/439590956 shutdown_connections 2024-03-25T19:20:07.109 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.107+0000 7f0d03954640 1 -- 172.21.15.83:0/439590956 >> 172.21.15.83:0/439590956 conn(0x7f0cfc0617a0 msgr2=0x7f0cfc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:07.109 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.108+0000 7f0d03954640 1 -- 172.21.15.83:0/439590956 shutdown_connections 2024-03-25T19:20:07.109 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.108+0000 7f0d03954640 1 -- 172.21.15.83:0/439590956 wait complete. 2024-03-25T19:20:07.111 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.109+0000 7f0d03954640 10 monclient: build_initial_monmap 2024-03-25T19:20:07.112 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.109+0000 7f0d03954640 10 monclient: monmap: 2024-03-25T19:20:07.112 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:20:07.112 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:20:07.112 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:20:07.112 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:20:07.112 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:07.112 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:20:07.112 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:20:07.112 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:20:07.112 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:20:07.112 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:20:07.112 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.109+0000 7f0d03954640 1 librados: starting msgr at 2024-03-25T19:20:07.112 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.109+0000 7f0d03954640 1 librados: starting objecter 2024-03-25T19:20:07.112 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.109+0000 7f0d03954640 1 Processor -- start 2024-03-25T19:20:07.112 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.109+0000 7f0d03954640 1 -- start start 2024-03-25T19:20:07.112 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.109+0000 7f0d03954640 1 librados: setting wanted keys 2024-03-25T19:20:07.113 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.109+0000 7f0d03954640 1 librados: calling monclient init 2024-03-25T19:20:07.113 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.109+0000 7f0d03954640 10 monclient: init 2024-03-25T19:20:07.113 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.110+0000 7f0d03954640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:07.113 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.110+0000 7f0d03954640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:20:07.113 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.110+0000 7f0d03954640 10 monclient(hunting): picked mon.noname-c con 0x7f0cfc0b1730 addr v1:172.21.15.83:6790/0 2024-03-25T19:20:07.113 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.110+0000 7f0d03954640 10 monclient(hunting): picked mon.noname-b con 0x7f0cfc14be90 addr v1:172.21.15.103:6789/0 2024-03-25T19:20:07.113 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.110+0000 7f0d03954640 10 monclient(hunting): picked mon.noname-a con 0x7f0cfc135e80 addr v1:172.21.15.83:6789/0 2024-03-25T19:20:07.113 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.110+0000 7f0d03954640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cfc137ee0 con 0x7f0cfc135e80 2024-03-25T19:20:07.113 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.110+0000 7f0d03954640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cfc1381c0 con 0x7f0cfc14be90 2024-03-25T19:20:07.113 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.110+0000 7f0d03954640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cfc136ff0 con 0x7f0cfc0b1730 2024-03-25T19:20:07.113 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.110+0000 7f0d03954640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:07.113 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.111+0000 7f0d03954640 10 monclient(hunting): authenticate will time out at 6064.382766s 2024-03-25T19:20:07.113 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.111+0000 7f0d01ef1640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f0cfc135e80 0x7f0cfc1e2480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:54380/0 (socket says 172.21.15.83:54380) 2024-03-25T19:20:07.113 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.111+0000 7f0d01ef1640 1 -- 172.21.15.83:0/781529790 learned_addr learned my addr 172.21.15.83:0/781529790 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:20:07.113 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.111+0000 7f0cf1ffb640 10 client.?.objecter ms_handle_connect 0x7f0cfc135e80 2024-03-25T19:20:07.114 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.111+0000 7f0cf1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:07.114 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.111+0000 7f0cf1ffb640 10 client.?.objecter ms_handle_connect 0x7f0cfc0b1730 2024-03-25T19:20:07.114 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.111+0000 7f0cf1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:07.114 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.111+0000 7f0cf1ffb640 10 client.?.objecter ms_handle_connect 0x7f0cfc14be90 2024-03-25T19:20:07.114 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.111+0000 7f0cf1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:07.116 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.113+0000 7f0cf1ffb640 1 -- 172.21.15.83:0/781529790 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1493019725 0 0) 0x7f0cfc136ff0 con 0x7f0cfc0b1730 2024-03-25T19:20:07.116 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.113+0000 7f0cf1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:20:07.116 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.113+0000 7f0cf1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:07.116 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.113+0000 7f0cf1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:07.116 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.113+0000 7f0cf1ffb640 10 monclient(hunting): my global_id is 4220 2024-03-25T19:20:07.116 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.114+0000 7f0cf1ffb640 1 -- 172.21.15.83:0/781529790 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ce0004eb0 con 0x7f0cfc0b1730 2024-03-25T19:20:07.116 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.114+0000 7f0cf1ffb640 1 -- 172.21.15.83:0/781529790 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 623830172 0 0) 0x7f0cfc1381c0 con 0x7f0cfc14be90 2024-03-25T19:20:07.116 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.114+0000 7f0cf1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:20:07.116 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.114+0000 7f0cf1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:07.116 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.114+0000 7f0cf1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:07.116 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.114+0000 7f0cf1ffb640 10 monclient(hunting): my global_id is 4216 2024-03-25T19:20:07.116 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.114+0000 7f0cf1ffb640 1 -- 172.21.15.83:0/781529790 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cfc136ff0 con 0x7f0cfc14be90 2024-03-25T19:20:07.116 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.114+0000 7f0cf1ffb640 1 -- 172.21.15.83:0/781529790 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 92255773 0 0) 0x7f0cfc137ee0 con 0x7f0cfc135e80 2024-03-25T19:20:07.116 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.114+0000 7f0cf1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:20:07.116 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.114+0000 7f0cf1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:07.116 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.114+0000 7f0cf1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:07.117 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.114+0000 7f0cf1ffb640 10 monclient(hunting): my global_id is 4209 2024-03-25T19:20:07.117 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.114+0000 7f0cf1ffb640 1 -- 172.21.15.83:0/781529790 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ce0005360 con 0x7f0cfc135e80 2024-03-25T19:20:07.121 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.120+0000 7f0cf1ffb640 1 -- 172.21.15.83:0/781529790 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4249739344 0 0) 0x7f0cfc136ff0 con 0x7f0cfc14be90 2024-03-25T19:20:07.121 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.120+0000 7f0cf1ffb640 1 -- 172.21.15.83:0/781529790 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0cfc137ee0 con 0x7f0cfc14be90 2024-03-25T19:20:07.121 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.120+0000 7f0cf1ffb640 1 -- 172.21.15.83:0/781529790 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f0cec0018b0 con 0x7f0cfc14be90 2024-03-25T19:20:07.121 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.120+0000 7f0cf1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:07.121 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.120+0000 7f0cf1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:20:07.121 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.120+0000 7f0cf1ffb640 10 monclient(hunting): dump: 2024-03-25T19:20:07.121 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:20:07.121 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:20:07.121 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:07.121 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:07.121 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:07.121 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:20:07.121 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:20:07.122 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:20:07.122 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:20:07.122 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:20:07.122 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.120+0000 7f0cf1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:20:07.122 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.120+0000 7f0cf1ffb640 1 -- 172.21.15.83:0/781529790 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3697685008 0 0) 0x7f0ce0005360 con 0x7f0cfc135e80 2024-03-25T19:20:07.122 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.121+0000 7f0cf1ffb640 1 -- 172.21.15.83:0/781529790 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ce0005cd0 con 0x7f0cfc135e80 2024-03-25T19:20:07.123 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.122+0000 7f0cf1ffb640 1 -- 172.21.15.83:0/781529790 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f0cf4001bd0 con 0x7f0cfc135e80 2024-03-25T19:20:07.123 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.122+0000 7f0cf1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:07.123 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.122+0000 7f0cf1ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:20:07.123 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.122+0000 7f0cf1ffb640 10 monclient(hunting): dump: 2024-03-25T19:20:07.123 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:20:07.123 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:20:07.123 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:07.123 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:07.123 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:07.123 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:20:07.124 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:20:07.124 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:20:07.124 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:20:07.124 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:20:07.124 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.123+0000 7f0cf1ffb640 1 -- 172.21.15.83:0/781529790 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2130644400 0 0) 0x7f0ce0004eb0 con 0x7f0cfc0b1730 2024-03-25T19:20:07.124 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.123+0000 7f0cf1ffb640 1 -- 172.21.15.83:0/781529790 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ce0005360 con 0x7f0cfc0b1730 2024-03-25T19:20:07.124 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.123+0000 7f0cf1ffb640 1 -- 172.21.15.83:0/781529790 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f0cf80047e0 con 0x7f0cfc0b1730 2024-03-25T19:20:07.124 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.123+0000 7f0cf1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:07.124 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.123+0000 7f0cf1ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:20:07.124 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.123+0000 7f0cf1ffb640 10 monclient(hunting): dump: 2024-03-25T19:20:07.124 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:20:07.124 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:20:07.124 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:07.124 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:07.124 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:07.124 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:20:07.125 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:20:07.125 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:20:07.125 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:20:07.125 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:20:07.128 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.126+0000 7f0cf1ffb640 1 -- 172.21.15.83:0/781529790 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3300838783 0 0) 0x7f0ce0005360 con 0x7f0cfc0b1730 2024-03-25T19:20:07.128 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.126+0000 7f0cf1ffb640 1 -- 172.21.15.83:0/781529790 >> v1:172.21.15.103:6789/0 conn(0x7f0cfc14be90 legacy=0x7f0cfc139120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:07.128 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.126+0000 7f0cf1ffb640 1 -- 172.21.15.83:0/781529790 >> v1:172.21.15.83:6789/0 conn(0x7f0cfc135e80 legacy=0x7f0cfc1e2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:07.128 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.126+0000 7f0cf1ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:20:07.128 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.126+0000 7f0cf1ffb640 1 monclient: found mon.c 2024-03-25T19:20:07.128 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.126+0000 7f0cf1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:07.128 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.126+0000 7f0cf1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:20:07.128 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.126+0000 7f0cf1ffb640 1 -- 172.21.15.83:0/781529790 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cfc136870 con 0x7f0cfc0b1730 2024-03-25T19:20:07.128 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.126+0000 7f0cf1ffb640 10 monclient: _finish_auth 0 2024-03-25T19:20:07.128 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.126+0000 7f0cf1ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:20:07.129 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.126+0000 7f0cf1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:07.129 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.127+0000 7f0d03954640 5 monclient: authenticate success, global_id 4220 2024-03-25T19:20:07.129 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.127+0000 7f0d03954640 10 monclient: _renew_subs 2024-03-25T19:20:07.129 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.127+0000 7f0d03954640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:20:07.129 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.127+0000 7f0d03954640 1 -- 172.21.15.83:0/781529790 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cfc1362e0 con 0x7f0cfc0b1730 2024-03-25T19:20:07.129 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.127+0000 7f0d03954640 10 client.4220.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:20:07.129 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.127+0000 7f0d03954640 10 monclient: _renew_subs 2024-03-25T19:20:07.129 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.127+0000 7f0d03954640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:20:07.129 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.127+0000 7f0d03954640 1 -- 172.21.15.83:0/781529790 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cfc1e3ba0 con 0x7f0cfc0b1730 2024-03-25T19:20:07.129 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.127+0000 7f0d03954640 1 librados: init done 2024-03-25T19:20:07.129 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.128+0000 7f0d03954640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:07.129 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.128+0000 7f0d03954640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:07.129 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.128+0000 7f0d03954640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:20:07.129 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.128+0000 7f0d03954640 1 -- 172.21.15.83:0/781529790 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cfc0b2270 con 0x7f0cfc0b1730 2024-03-25T19:20:07.139 INFO:tasks.ceph.osd.7.smithi103.stderr:==00:00:00:16.626 35635== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:07.289 INFO:tasks.ceph.osd.4.smithi103.stderr:==00:00:00:16.799 35632== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:07.322 INFO:tasks.ceph.osd.6.smithi103.stderr:==00:00:00:16.807 35634== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:07.411 INFO:tasks.ceph.osd.1.smithi083.stderr:==00:00:00:16.906 35987== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:07.448 INFO:tasks.ceph.osd.3.smithi083.stderr:==00:00:00:16.932 35989== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:07.474 INFO:tasks.ceph.osd.0.smithi083.stderr:==00:00:00:16.970 35986== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:07.584 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.584+0000 7f0cf1ffb640 1 -- 172.21.15.83:0/781529790 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0cf8001100 con 0x7f0cfc0b1730 2024-03-25T19:20:07.584 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.584+0000 7f0cf1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:07.584 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.584+0000 7f0cf1ffb640 1 -- 172.21.15.83:0/781529790 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f0cf8004d90 con 0x7f0cfc0b1730 2024-03-25T19:20:07.584 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.584+0000 7f0cf1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:07.584 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.584+0000 7f0cf1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:20:07.584 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.584+0000 7f0cf1ffb640 10 monclient: dump: 2024-03-25T19:20:07.584 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:20:07.585 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:20:07.585 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:07.585 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:07.585 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:07.585 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:20:07.585 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:20:07.585 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:20:07.585 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:20:07.585 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:20:07.585 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.584+0000 7f0cf37fe640 4 set_mon_vals no callback set 2024-03-25T19:20:07.585 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.586+0000 7f0cf1ffb640 1 -- 172.21.15.83:0/781529790 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7f0cf801ad60 con 0x7f0cfc0b1730 2024-03-25T19:20:07.586 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.586+0000 7f0d00eef640 10 monclient: get_auth_request con 0x7f0ce006e730 auth_method 0 2024-03-25T19:20:07.586 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.587+0000 7f0cf1ffb640 10 client.4220.objecter ms_handle_connect 0x7f0ce006e730 2024-03-25T19:20:07.590 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.590+0000 7f0cf1ffb640 1 -- 172.21.15.83:0/781529790 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7f0cf80548a0 con 0x7f0cfc0b1730 2024-03-25T19:20:07.590 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.590+0000 7f0cf1ffb640 10 client.4220.objecter ms_dispatch 0x7f0cfc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:20:07.590 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.590+0000 7f0cf1ffb640 3 client.4220.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:20:07.590 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.590+0000 7f0cf1ffb640 3 client.4220.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:20:07.590 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.590+0000 7f0cf1ffb640 20 client.4220.objecter dump_active .. 0 homeless 2024-03-25T19:20:07.590 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.590+0000 7f0cf1ffb640 1 -- 172.21.15.83:0/781529790 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0cf80501e0 con 0x7f0cfc0b1730 2024-03-25T19:20:07.590 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.590+0000 7f0cf1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:07.590 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.590+0000 7f0cf1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:20:07.803 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.803+0000 7f0d03954640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:07.803 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.803+0000 7f0d03954640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:07.803 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.803+0000 7f0d03954640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:20:07.803 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.803+0000 7f0d03954640 1 -- 172.21.15.83:0/781529790 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0cfc0627b0 con 0x7f0cfc0b1730 2024-03-25T19:20:07.861 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.861+0000 7f0cf1ffb640 1 -- 172.21.15.83:0/781529790 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7f0cf80522e0 con 0x7f0cfc0b1730 2024-03-25T19:20:07.861 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.861+0000 7f0cf1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:07.861 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.861+0000 7f0cf1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:20:07.862 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:20:07.862 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:20:07.864 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.864+0000 7f0d03954640 10 librados: watch_flush enter 2024-03-25T19:20:07.864 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.864+0000 7f0d03954640 10 librados: watch_flush exit 2024-03-25T19:20:07.864 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.864+0000 7f0d03954640 20 client.4220.objecter shutdown clearing up homeless session... 2024-03-25T19:20:07.864 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.864+0000 7f0d03954640 10 client.4220.objecter successfully canceled tick 2024-03-25T19:20:07.865 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.865+0000 7f0d03954640 1 -- 172.21.15.83:0/781529790 >> v1:172.21.15.103:6800/345072327 conn(0x7f0ce006e730 legacy=0x7f0ce0070bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:07.865 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.865+0000 7f0d03954640 10 monclient: shutdown 2024-03-25T19:20:07.865 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.865+0000 7f0d03954640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:07.865 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.865+0000 7f0d03954640 1 -- 172.21.15.83:0/781529790 >> v1:172.21.15.83:6790/0 conn(0x7f0cfc0b1730 legacy=0x7f0cfc144bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:07.865 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.865+0000 7f0d03954640 1 -- 172.21.15.83:0/781529790 shutdown_connections 2024-03-25T19:20:07.865 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.865+0000 7f0d03954640 1 -- 172.21.15.83:0/781529790 >> 172.21.15.83:0/781529790 conn(0x7f0cfc0617a0 msgr2=0x7f0cfc13c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:07.865 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.865+0000 7f0d03954640 1 -- 172.21.15.83:0/781529790 shutdown_connections 2024-03-25T19:20:07.865 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.865+0000 7f0d03954640 1 -- 172.21.15.83:0/781529790 wait complete. 2024-03-25T19:20:07.865 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:07.865+0000 7f0d03954640 1 librados: shutdown 2024-03-25T19:20:07.875 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:20:07.965 INFO:tasks.ceph.osd.2.smithi083.stderr:==00:00:00:17.457 35988== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:12.786 INFO:tasks.ceph.osd.5.smithi103.stderr:==00:00:00:22.278 35633== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:12.789 INFO:tasks.ceph.osd.5.smithi103.stderr:==00:00:00:22.282 35633== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:13.051 INFO:tasks.ceph.osd.6.smithi103.stderr:==00:00:00:22.536 35634== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:13.055 INFO:tasks.ceph.osd.6.smithi103.stderr:==00:00:00:22.540 35634== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:13.126 INFO:tasks.ceph.osd.1.smithi083.stderr:==00:00:00:22.621 35987== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:13.129 INFO:tasks.ceph.osd.1.smithi083.stderr:==00:00:00:22.625 35987== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:13.157 INFO:tasks.ceph.osd.7.smithi103.stderr:==00:00:00:22.643 35635== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:13.162 INFO:tasks.ceph.osd.7.smithi103.stderr:==00:00:00:22.648 35635== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:13.205 INFO:tasks.ceph.osd.4.smithi103.stderr:==00:00:00:22.714 35632== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:13.210 INFO:tasks.ceph.osd.4.smithi103.stderr:==00:00:00:22.719 35632== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:13.212 INFO:tasks.ceph.osd.0.smithi083.stderr:==00:00:00:22.708 35986== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:13.216 INFO:tasks.ceph.osd.0.smithi083.stderr:==00:00:00:22.712 35986== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:13.492 INFO:tasks.ceph.osd.3.smithi083.stderr:==00:00:00:22.977 35989== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:13.495 INFO:tasks.ceph.osd.3.smithi083.stderr:==00:00:00:22.980 35989== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:13.786 INFO:tasks.ceph.osd.2.smithi083.stderr:==00:00:00:23.278 35988== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:13.791 INFO:tasks.ceph.osd.2.smithi083.stderr:==00:00:00:23.283 35988== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:14.681 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:20:14.773 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.773+0000 7f879fc45640 10 monclient: get_monmap_and_config 2024-03-25T19:20:14.773 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.773+0000 7f879fc45640 10 monclient: build_initial_monmap 2024-03-25T19:20:14.774 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.773+0000 7f879fc45640 10 monclient: monmap: 2024-03-25T19:20:14.774 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:20:14.774 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:20:14.774 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:20:14.774716+0000 2024-03-25T19:20:14.774 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:20:14.774716+0000 2024-03-25T19:20:14.774 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:14.774 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:20:14.774 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:20:14.774 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:20:14.774 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:20:14.774 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:20:14.774 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.774+0000 7f879fc45640 1 Processor -- start 2024-03-25T19:20:14.776 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.774+0000 7f879fc45640 1 -- start start 2024-03-25T19:20:14.776 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.774+0000 7f879fc45640 10 monclient: init 2024-03-25T19:20:14.776 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.775+0000 7f879fc45640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:14.776 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.775+0000 7f879fc45640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:20:14.776 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.775+0000 7f879fc45640 10 monclient(hunting): picked mon.noname-b con 0x7f87980b4d60 addr v1:172.21.15.83:6790/0 2024-03-25T19:20:14.776 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.775+0000 7f879fc45640 10 monclient(hunting): picked mon.noname-a con 0x7f87980b1730 addr v1:172.21.15.83:6789/0 2024-03-25T19:20:14.777 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.775+0000 7f879fc45640 10 monclient(hunting): picked mon.noname-c con 0x7f879814be90 addr v1:172.21.15.103:6789/0 2024-03-25T19:20:14.777 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.775+0000 7f879fc45640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87980af940 con 0x7f87980b1730 2024-03-25T19:20:14.777 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.775+0000 7f879fc45640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87980b0a70 con 0x7f879814be90 2024-03-25T19:20:14.777 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.775+0000 7f879fc45640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87980b0d50 con 0x7f87980b4d60 2024-03-25T19:20:14.777 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.775+0000 7f879fc45640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:14.777 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.775+0000 7f879fc45640 10 monclient(hunting): authenticate will time out at 6072.047215s 2024-03-25T19:20:14.777 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.775+0000 7f879d1e0640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f87980b1730 0x7f87980b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:54382/0 (socket says 172.21.15.83:54382) 2024-03-25T19:20:14.777 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.775+0000 7f879d1e0640 1 -- 172.21.15.83:0/2418249307 learned_addr learned my addr 172.21.15.83:0/2418249307 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:20:14.778 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.778+0000 7f879c9df640 1 -- 172.21.15.83:0/2418249307 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2901818485 0 0) 0x7f87980af940 con 0x7f87980b1730 2024-03-25T19:20:14.778 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.778+0000 7f879c9df640 10 monclient(hunting): _negotiate 2024-03-25T19:20:14.778 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.778+0000 7f879c9df640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:14.778 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.778+0000 7f879c9df640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:14.778 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.778+0000 7f879c9df640 10 monclient(hunting): my global_id is 4212 2024-03-25T19:20:14.778 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.778+0000 7f879c9df640 1 -- 172.21.15.83:0/2418249307 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8784003520 con 0x7f87980b1730 2024-03-25T19:20:14.780 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.779+0000 7f879c9df640 1 -- 172.21.15.83:0/2418249307 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3626766381 0 0) 0x7f87980b0d50 con 0x7f87980b4d60 2024-03-25T19:20:14.780 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.779+0000 7f879c9df640 10 monclient(hunting): _negotiate 2024-03-25T19:20:14.780 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.779+0000 7f879c9df640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:14.780 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.779+0000 7f879c9df640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:14.780 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.779+0000 7f879c9df640 10 monclient(hunting): my global_id is 4223 2024-03-25T19:20:14.780 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.779+0000 7f879c9df640 1 -- 172.21.15.83:0/2418249307 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87980af940 con 0x7f87980b4d60 2024-03-25T19:20:14.780 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.779+0000 7f879c9df640 1 -- 172.21.15.83:0/2418249307 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 656107463 0 0) 0x7f87980b0a70 con 0x7f879814be90 2024-03-25T19:20:14.780 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.779+0000 7f879c9df640 10 monclient(hunting): _negotiate 2024-03-25T19:20:14.780 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.779+0000 7f879c9df640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:14.781 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.779+0000 7f879c9df640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:14.781 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.779+0000 7f879c9df640 10 monclient(hunting): my global_id is 4219 2024-03-25T19:20:14.781 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.779+0000 7f879c9df640 1 -- 172.21.15.83:0/2418249307 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87980b0d50 con 0x7f879814be90 2024-03-25T19:20:14.783 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.781+0000 7f879c9df640 1 -- 172.21.15.83:0/2418249307 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2025280505 0 0) 0x7f8784003520 con 0x7f87980b1730 2024-03-25T19:20:14.783 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.781+0000 7f879c9df640 1 -- 172.21.15.83:0/2418249307 >> v1:172.21.15.83:6790/0 conn(0x7f87980b4d60 legacy=0x7f879813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:14.783 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.781+0000 7f879c9df640 1 -- 172.21.15.83:0/2418249307 >> v1:172.21.15.103:6789/0 conn(0x7f879814be90 legacy=0x7f879814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:14.783 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.781+0000 7f879c9df640 10 monclient: _finish_hunting 0 2024-03-25T19:20:14.783 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.781+0000 7f879c9df640 1 monclient: found mon.noname-a 2024-03-25T19:20:14.783 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.781+0000 7f879c9df640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:14.783 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.781+0000 7f879c9df640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.83:6789/0 2024-03-25T19:20:14.783 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.781+0000 7f879c9df640 1 -- 172.21.15.83:0/2418249307 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87980b1030 con 0x7f87980b1730 2024-03-25T19:20:14.783 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.781+0000 7f879c9df640 10 monclient: _finish_auth 0 2024-03-25T19:20:14.783 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.781+0000 7f879c9df640 10 monclient: _check_auth_tickets 2024-03-25T19:20:14.783 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.781+0000 7f879c9df640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:14.783 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.781+0000 7f879c9df640 1 -- 172.21.15.83:0/2418249307 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f8788001690 con 0x7f87980b1730 2024-03-25T19:20:14.783 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.781+0000 7f879c9df640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:14.783 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.781+0000 7f879c9df640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:20:14.784 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.781+0000 7f879c9df640 10 monclient: dump: 2024-03-25T19:20:14.784 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:20:14.784 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:20:14.784 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:14.784 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:14.784 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:14.784 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:20:14.784 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:20:14.784 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:20:14.784 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:20:14.784 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:20:14.784 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.781+0000 7f879fc45640 5 monclient: authenticate success, global_id 4212 2024-03-25T19:20:14.784 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.781+0000 7f879fc45640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:20:14.786 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.783+0000 7f879c9df640 1 -- 172.21.15.83:0/2418249307 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f87880017f0 con 0x7f87980b1730 2024-03-25T19:20:14.786 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.783+0000 7f879c9df640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:14.786 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.783+0000 7f879c9df640 1 -- 172.21.15.83:0/2418249307 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f8788001c00 con 0x7f87980b1730 2024-03-25T19:20:14.786 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.783+0000 7f879c9df640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:14.786 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.783+0000 7f879c9df640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:20:14.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.783+0000 7f879c9df640 10 monclient: dump: 2024-03-25T19:20:14.787 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:20:14.787 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:20:14.787 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:14.787 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:14.787 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:14.787 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:20:14.787 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:20:14.787 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:20:14.787 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:20:14.787 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:20:14.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.783+0000 7f879fc45640 10 monclient: get_monmap_and_config success 2024-03-25T19:20:14.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.783+0000 7f879fc45640 4 set_mon_vals no callback set 2024-03-25T19:20:14.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.784+0000 7f879fc45640 10 monclient: shutdown 2024-03-25T19:20:14.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.784+0000 7f879fc45640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:14.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.784+0000 7f879fc45640 1 -- 172.21.15.83:0/2418249307 >> v1:172.21.15.83:6789/0 conn(0x7f87980b1730 legacy=0x7f87980b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:14.788 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.784+0000 7f879fc45640 1 -- 172.21.15.83:0/2418249307 shutdown_connections 2024-03-25T19:20:14.788 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.784+0000 7f879fc45640 1 -- 172.21.15.83:0/2418249307 >> 172.21.15.83:0/2418249307 conn(0x7f87980617a0 msgr2=0x7f8798061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:14.788 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.785+0000 7f879fc45640 1 -- 172.21.15.83:0/2418249307 shutdown_connections 2024-03-25T19:20:14.788 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.785+0000 7f879fc45640 1 -- 172.21.15.83:0/2418249307 wait complete. 2024-03-25T19:20:14.788 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.786+0000 7f879fc45640 10 monclient: build_initial_monmap 2024-03-25T19:20:14.788 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.786+0000 7f879fc45640 10 monclient: monmap: 2024-03-25T19:20:14.788 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:20:14.788 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:20:14.788 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:20:14.788 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:20:14.788 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:14.788 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:20:14.788 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:20:14.788 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:20:14.789 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:20:14.789 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:20:14.789 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.786+0000 7f879fc45640 1 librados: starting msgr at 2024-03-25T19:20:14.789 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.786+0000 7f879fc45640 1 librados: starting objecter 2024-03-25T19:20:14.789 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.786+0000 7f879fc45640 1 Processor -- start 2024-03-25T19:20:14.789 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.786+0000 7f879fc45640 1 -- start start 2024-03-25T19:20:14.789 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.786+0000 7f879fc45640 1 librados: setting wanted keys 2024-03-25T19:20:14.789 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.786+0000 7f879fc45640 1 librados: calling monclient init 2024-03-25T19:20:14.789 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.786+0000 7f879fc45640 10 monclient: init 2024-03-25T19:20:14.791 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.787+0000 7f879fc45640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:14.791 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.787+0000 7f879fc45640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:20:14.791 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.787+0000 7f879fc45640 10 monclient(hunting): picked mon.noname-c con 0x7f87980b1730 addr v1:172.21.15.83:6790/0 2024-03-25T19:20:14.791 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.787+0000 7f879fc45640 10 monclient(hunting): picked mon.noname-b con 0x7f87980b4d60 addr v1:172.21.15.103:6789/0 2024-03-25T19:20:14.791 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.788+0000 7f879fc45640 10 monclient(hunting): picked mon.noname-a con 0x7f879814be90 addr v1:172.21.15.83:6789/0 2024-03-25T19:20:14.791 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.788+0000 7f879fc45640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8798141f10 con 0x7f879814be90 2024-03-25T19:20:14.791 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.788+0000 7f879fc45640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8798140da0 con 0x7f87980b4d60 2024-03-25T19:20:14.791 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.788+0000 7f879fc45640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8798138860 con 0x7f87980b1730 2024-03-25T19:20:14.791 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.788+0000 7f879fc45640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:14.791 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.788+0000 7f879fc45640 10 monclient(hunting): authenticate will time out at 6072.060166s 2024-03-25T19:20:14.791 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.788+0000 7f879e1e2640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f879814be90 0x7f87981e2140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:54392/0 (socket says 172.21.15.83:54392) 2024-03-25T19:20:14.792 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.788+0000 7f879e1e2640 1 -- 172.21.15.83:0/2641445452 learned_addr learned my addr 172.21.15.83:0/2641445452 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:20:14.792 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.788+0000 7f878e7fc640 10 client.?.objecter ms_handle_connect 0x7f879814be90 2024-03-25T19:20:14.792 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.788+0000 7f878e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:14.792 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.789+0000 7f878e7fc640 10 client.?.objecter ms_handle_connect 0x7f87980b4d60 2024-03-25T19:20:14.792 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.789+0000 7f878e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:14.792 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.789+0000 7f878e7fc640 10 client.?.objecter ms_handle_connect 0x7f87980b1730 2024-03-25T19:20:14.792 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.789+0000 7f878e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:14.792 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.790+0000 7f878e7fc640 1 -- 172.21.15.83:0/2641445452 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2279425904 0 0) 0x7f8798141f10 con 0x7f879814be90 2024-03-25T19:20:14.792 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.790+0000 7f878e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:20:14.792 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.790+0000 7f878e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:14.792 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.790+0000 7f878e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:14.792 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.790+0000 7f878e7fc640 10 monclient(hunting): my global_id is 4215 2024-03-25T19:20:14.792 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.790+0000 7f878e7fc640 1 -- 172.21.15.83:0/2641445452 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f877c004eb0 con 0x7f879814be90 2024-03-25T19:20:14.794 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.792+0000 7f878e7fc640 1 -- 172.21.15.83:0/2641445452 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1167100486 0 0) 0x7f8798138860 con 0x7f87980b1730 2024-03-25T19:20:14.794 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.792+0000 7f878e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:20:14.794 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.792+0000 7f878e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:14.794 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.792+0000 7f878e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:14.794 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.792+0000 7f878e7fc640 10 monclient(hunting): my global_id is 4226 2024-03-25T19:20:14.794 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.792+0000 7f878e7fc640 1 -- 172.21.15.83:0/2641445452 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8798141f10 con 0x7f87980b1730 2024-03-25T19:20:14.794 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.792+0000 7f878e7fc640 1 -- 172.21.15.83:0/2641445452 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2854011757 0 0) 0x7f8798140da0 con 0x7f87980b4d60 2024-03-25T19:20:14.794 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.792+0000 7f878e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:20:14.794 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.792+0000 7f878e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:14.794 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.792+0000 7f878e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:14.794 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.792+0000 7f878e7fc640 10 monclient(hunting): my global_id is 4222 2024-03-25T19:20:14.794 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.792+0000 7f878e7fc640 1 -- 172.21.15.83:0/2641445452 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8798138860 con 0x7f87980b4d60 2024-03-25T19:20:14.795 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.795+0000 7f878e7fc640 1 -- 172.21.15.83:0/2641445452 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 585702252 0 0) 0x7f877c004eb0 con 0x7f879814be90 2024-03-25T19:20:14.795 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.795+0000 7f878e7fc640 1 -- 172.21.15.83:0/2641445452 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8798140da0 con 0x7f879814be90 2024-03-25T19:20:14.795 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.795+0000 7f878e7fc640 1 -- 172.21.15.83:0/2641445452 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f8790001c10 con 0x7f879814be90 2024-03-25T19:20:14.795 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.795+0000 7f878e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:14.795 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.795+0000 7f878e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:20:14.795 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.795+0000 7f878e7fc640 10 monclient(hunting): dump: 2024-03-25T19:20:14.795 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:20:14.795 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:20:14.795 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:14.795 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:14.795 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:14.795 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:20:14.796 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:20:14.796 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:20:14.796 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:20:14.796 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:20:14.796 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.795+0000 7f878e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:20:14.799 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.798+0000 7f878e7fc640 1 -- 172.21.15.83:0/2641445452 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2295023795 0 0) 0x7f8798140da0 con 0x7f879814be90 2024-03-25T19:20:14.799 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.798+0000 7f878e7fc640 1 -- 172.21.15.83:0/2641445452 >> v1:172.21.15.83:6790/0 conn(0x7f87980b1730 legacy=0x7f87981405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:14.799 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.798+0000 7f878e7fc640 1 -- 172.21.15.83:0/2641445452 >> v1:172.21.15.103:6789/0 conn(0x7f87980b4d60 legacy=0x7f87981d7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:14.799 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.798+0000 7f878e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:20:14.799 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.798+0000 7f878e7fc640 1 monclient: found mon.a 2024-03-25T19:20:14.799 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.799+0000 7f878e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:14.799 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.799+0000 7f878e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:20:14.799 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.799+0000 7f878e7fc640 1 -- 172.21.15.83:0/2641445452 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8798137f50 con 0x7f879814be90 2024-03-25T19:20:14.799 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.799+0000 7f878e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:20:14.800 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.799+0000 7f878e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:20:14.800 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.799+0000 7f878e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:14.800 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.799+0000 7f879fc45640 5 monclient: authenticate success, global_id 4215 2024-03-25T19:20:14.800 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.799+0000 7f879fc45640 10 monclient: _renew_subs 2024-03-25T19:20:14.800 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.799+0000 7f879fc45640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:20:14.800 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.799+0000 7f879fc45640 1 -- 172.21.15.83:0/2641445452 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87981d8080 con 0x7f879814be90 2024-03-25T19:20:14.800 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.799+0000 7f879fc45640 10 client.4215.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:20:14.800 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.799+0000 7f879fc45640 10 monclient: _renew_subs 2024-03-25T19:20:14.800 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.799+0000 7f879fc45640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:20:14.800 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.799+0000 7f879fc45640 1 -- 172.21.15.83:0/2641445452 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f87981d8560 con 0x7f879814be90 2024-03-25T19:20:14.800 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.799+0000 7f879fc45640 1 librados: init done 2024-03-25T19:20:14.801 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.801+0000 7f879fc45640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:14.801 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.801+0000 7f879fc45640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:14.801 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.801+0000 7f879fc45640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:20:14.801 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.801+0000 7f879fc45640 1 -- 172.21.15.83:0/2641445452 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87981381a0 con 0x7f879814be90 2024-03-25T19:20:14.819 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.817+0000 7f878e7fc640 1 -- 172.21.15.83:0/2641445452 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8790003970 con 0x7f879814be90 2024-03-25T19:20:14.819 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.817+0000 7f878e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:14.819 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.818+0000 7f878e7fc640 1 -- 172.21.15.83:0/2641445452 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f8790002100 con 0x7f879814be90 2024-03-25T19:20:14.819 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.818+0000 7f878e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:14.819 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.818+0000 7f878e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:20:14.819 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.818+0000 7f878e7fc640 10 monclient: dump: 2024-03-25T19:20:14.819 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:20:14.819 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:20:14.819 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:14.819 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:14.819 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:14.819 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:20:14.819 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:20:14.820 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:20:14.820 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:20:14.820 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:20:14.820 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.818+0000 7f878ffff640 4 set_mon_vals no callback set 2024-03-25T19:20:14.820 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.818+0000 7f878e7fc640 1 -- 172.21.15.83:0/2641445452 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7f8790003e00 con 0x7f879814be90 2024-03-25T19:20:14.820 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.819+0000 7f879d9e1640 10 monclient: get_auth_request con 0x7f877c06de50 auth_method 0 2024-03-25T19:20:14.820 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.819+0000 7f878e7fc640 10 client.4215.objecter ms_handle_connect 0x7f877c06de50 2024-03-25T19:20:14.821 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.820+0000 7f878e7fc640 1 -- 172.21.15.83:0/2641445452 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7f8790054b20 con 0x7f879814be90 2024-03-25T19:20:14.821 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.820+0000 7f878e7fc640 10 client.4215.objecter ms_dispatch 0x7f8798000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:20:14.821 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.820+0000 7f878e7fc640 3 client.4215.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:20:14.821 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.820+0000 7f878e7fc640 3 client.4215.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:20:14.821 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:14.820+0000 7f878e7fc640 20 client.4215.objecter dump_active .. 0 homeless 2024-03-25T19:20:15.074 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:15.074+0000 7f878e7fc640 1 -- 172.21.15.83:0/2641445452 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8790050630 con 0x7f879814be90 2024-03-25T19:20:15.074 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:15.074+0000 7f878e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:15.074 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:15.074+0000 7f878e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:20:15.321 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:15.321+0000 7f879fc45640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:15.321 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:15.321+0000 7f879fc45640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:15.321 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:15.321+0000 7f879fc45640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:20:15.321 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:15.321+0000 7f879fc45640 1 -- 172.21.15.83:0/2641445452 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f87980627b0 con 0x7f879814be90 2024-03-25T19:20:15.345 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:15.346+0000 7f878e7fc640 1 -- 172.21.15.83:0/2641445452 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7f8790052560 con 0x7f879814be90 2024-03-25T19:20:15.346 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:15.346+0000 7f878e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:15.346 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:15.346+0000 7f878e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:20:15.346 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:20:15.346 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:20:15.349 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:15.348+0000 7f879fc45640 10 librados: watch_flush enter 2024-03-25T19:20:15.349 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:15.348+0000 7f879fc45640 10 librados: watch_flush exit 2024-03-25T19:20:15.349 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:15.348+0000 7f879fc45640 20 client.4215.objecter shutdown clearing up homeless session... 2024-03-25T19:20:15.349 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:15.348+0000 7f879fc45640 10 client.4215.objecter successfully canceled tick 2024-03-25T19:20:15.349 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:15.348+0000 7f879fc45640 1 -- 172.21.15.83:0/2641445452 >> v1:172.21.15.103:6800/345072327 conn(0x7f877c06de50 legacy=0x7f877c070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:15.349 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:15.349+0000 7f879fc45640 10 monclient: shutdown 2024-03-25T19:20:15.349 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:15.349+0000 7f879fc45640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:15.349 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:15.349+0000 7f879fc45640 1 -- 172.21.15.83:0/2641445452 >> v1:172.21.15.83:6789/0 conn(0x7f879814be90 legacy=0x7f87981e2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:15.349 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:15.349+0000 7f879fc45640 1 -- 172.21.15.83:0/2641445452 shutdown_connections 2024-03-25T19:20:15.349 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:15.349+0000 7f879fc45640 1 -- 172.21.15.83:0/2641445452 >> 172.21.15.83:0/2641445452 conn(0x7f87980617a0 msgr2=0x7f87980b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:15.349 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:15.349+0000 7f879fc45640 1 -- 172.21.15.83:0/2641445452 shutdown_connections 2024-03-25T19:20:15.350 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:15.349+0000 7f879fc45640 1 -- 172.21.15.83:0/2641445452 wait complete. 2024-03-25T19:20:15.350 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:15.349+0000 7f879fc45640 1 librados: shutdown 2024-03-25T19:20:15.359 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:20:16.695 INFO:tasks.ceph.osd.5.smithi103.stderr:==00:00:00:26.187 35633== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:17.147 INFO:tasks.ceph.osd.4.smithi103.stderr:==00:00:00:26.656 35632== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:17.247 INFO:tasks.ceph.osd.6.smithi103.stderr:==00:00:00:26.732 35634== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:17.271 INFO:tasks.ceph.osd.7.smithi103.stderr:==00:00:00:26.757 35635== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:17.364 INFO:tasks.ceph.osd.1.smithi083.stderr:==00:00:00:26.859 35987== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:17.723 INFO:tasks.ceph.osd.0.smithi083.stderr:==00:00:00:27.219 35986== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:18.009 INFO:tasks.ceph.osd.2.smithi083.stderr:==00:00:00:27.501 35988== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:18.014 INFO:tasks.ceph.osd.3.smithi083.stderr:==00:00:00:27.499 35989== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:22.164 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:20:22.253 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.254+0000 7f20f6b9d640 10 monclient: get_monmap_and_config 2024-03-25T19:20:22.254 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.254+0000 7f20f6b9d640 10 monclient: build_initial_monmap 2024-03-25T19:20:22.254 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.254+0000 7f20f6b9d640 10 monclient: monmap: 2024-03-25T19:20:22.254 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:20:22.254 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:20:22.254 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:20:22.255056+0000 2024-03-25T19:20:22.254 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:20:22.255056+0000 2024-03-25T19:20:22.254 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:22.254 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:20:22.254 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:20:22.254 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:20:22.254 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:20:22.254 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:20:22.255 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.254+0000 7f20f6b9d640 1 Processor -- start 2024-03-25T19:20:22.256 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.254+0000 7f20f6b9d640 1 -- start start 2024-03-25T19:20:22.256 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.254+0000 7f20f6b9d640 10 monclient: init 2024-03-25T19:20:22.256 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.255+0000 7f20f6b9d640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:22.256 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.255+0000 7f20f6b9d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:20:22.256 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.255+0000 7f20f6b9d640 10 monclient(hunting): picked mon.noname-c con 0x7f20f00b4d60 addr v1:172.21.15.103:6789/0 2024-03-25T19:20:22.256 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.255+0000 7f20f6b9d640 10 monclient(hunting): picked mon.noname-a con 0x7f20f00b1730 addr v1:172.21.15.83:6789/0 2024-03-25T19:20:22.256 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.255+0000 7f20f6b9d640 10 monclient(hunting): picked mon.noname-b con 0x7f20f014be90 addr v1:172.21.15.83:6790/0 2024-03-25T19:20:22.256 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.255+0000 7f20f6b9d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20f00af940 con 0x7f20f00b1730 2024-03-25T19:20:22.256 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.255+0000 7f20f6b9d640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20f00b0a70 con 0x7f20f00b4d60 2024-03-25T19:20:22.256 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.255+0000 7f20f6b9d640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20f00b0d50 con 0x7f20f014be90 2024-03-25T19:20:22.256 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.255+0000 7f20f6b9d640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:22.256 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.255+0000 7f20f6b9d640 10 monclient(hunting): authenticate will time out at 6079.527356s 2024-03-25T19:20:22.256 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.256+0000 7f20effff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f20f00b1730 0x7f20f00b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:47998/0 (socket says 172.21.15.83:47998) 2024-03-25T19:20:22.256 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.256+0000 7f20effff640 1 -- 172.21.15.83:0/4253289674 learned_addr learned my addr 172.21.15.83:0/4253289674 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:20:22.258 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.259+0000 7f20ef7fe640 1 -- 172.21.15.83:0/4253289674 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4230107921 0 0) 0x7f20f00af940 con 0x7f20f00b1730 2024-03-25T19:20:22.258 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.259+0000 7f20ef7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:20:22.259 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.259+0000 7f20ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:22.259 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.259+0000 7f20ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:22.259 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.259+0000 7f20ef7fe640 10 monclient(hunting): my global_id is 4218 2024-03-25T19:20:22.259 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.259+0000 7f20ef7fe640 1 -- 172.21.15.83:0/4253289674 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20e0003520 con 0x7f20f00b1730 2024-03-25T19:20:22.260 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.259+0000 7f20ef7fe640 1 -- 172.21.15.83:0/4253289674 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2140207564 0 0) 0x7f20f00b0a70 con 0x7f20f00b4d60 2024-03-25T19:20:22.260 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.259+0000 7f20ef7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:20:22.260 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.259+0000 7f20ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:22.260 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.259+0000 7f20ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:22.260 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.259+0000 7f20ef7fe640 10 monclient(hunting): my global_id is 4225 2024-03-25T19:20:22.260 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.259+0000 7f20ef7fe640 1 -- 172.21.15.83:0/4253289674 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20f00af940 con 0x7f20f00b4d60 2024-03-25T19:20:22.260 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.259+0000 7f20ef7fe640 1 -- 172.21.15.83:0/4253289674 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 131991287 0 0) 0x7f20f00b0d50 con 0x7f20f014be90 2024-03-25T19:20:22.260 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.259+0000 7f20ef7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:20:22.260 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.259+0000 7f20ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:22.260 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.259+0000 7f20ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:22.260 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.259+0000 7f20ef7fe640 10 monclient(hunting): my global_id is 4229 2024-03-25T19:20:22.260 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.259+0000 7f20ef7fe640 1 -- 172.21.15.83:0/4253289674 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20e00039d0 con 0x7f20f014be90 2024-03-25T19:20:22.263 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.263+0000 7f20ef7fe640 1 -- 172.21.15.83:0/4253289674 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2086302427 0 0) 0x7f20f00af940 con 0x7f20f00b4d60 2024-03-25T19:20:22.263 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.263+0000 7f20ef7fe640 1 -- 172.21.15.83:0/4253289674 >> v1:172.21.15.83:6790/0 conn(0x7f20f014be90 legacy=0x7f20f014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:22.263 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.263+0000 7f20ef7fe640 1 -- 172.21.15.83:0/4253289674 >> v1:172.21.15.83:6789/0 conn(0x7f20f00b1730 legacy=0x7f20f00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:22.263 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.263+0000 7f20ef7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:20:22.263 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.263+0000 7f20ef7fe640 1 monclient: found mon.noname-c 2024-03-25T19:20:22.263 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.263+0000 7f20ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:22.263 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.263+0000 7f20ef7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.103:6789/0 2024-03-25T19:20:22.263 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.263+0000 7f20ef7fe640 1 -- 172.21.15.83:0/4253289674 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20f00b1030 con 0x7f20f00b4d60 2024-03-25T19:20:22.264 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.263+0000 7f20ef7fe640 10 monclient: _finish_auth 0 2024-03-25T19:20:22.264 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.263+0000 7f20ef7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:20:22.264 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.263+0000 7f20ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:22.264 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.263+0000 7f20ef7fe640 1 -- 172.21.15.83:0/4253289674 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f20e40015f0 con 0x7f20f00b4d60 2024-03-25T19:20:22.264 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.263+0000 7f20ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:22.264 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.263+0000 7f20ef7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:20:22.264 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.263+0000 7f20ef7fe640 10 monclient: dump: 2024-03-25T19:20:22.264 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:20:22.264 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:20:22.264 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:22.264 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:22.264 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:22.264 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:20:22.264 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:20:22.264 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:20:22.264 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:20:22.265 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:20:22.265 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.263+0000 7f20f6b9d640 5 monclient: authenticate success, global_id 4225 2024-03-25T19:20:22.265 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.263+0000 7f20f6b9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:20:22.267 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.265+0000 7f20ef7fe640 1 -- 172.21.15.83:0/4253289674 <== mon.1 v1:172.21.15.103:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20e4001750 con 0x7f20f00b4d60 2024-03-25T19:20:22.267 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.265+0000 7f20ef7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:22.267 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.265+0000 7f20ef7fe640 1 -- 172.21.15.83:0/4253289674 <== mon.1 v1:172.21.15.103:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f20e4001ba0 con 0x7f20f00b4d60 2024-03-25T19:20:22.267 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.265+0000 7f20ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:22.267 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.265+0000 7f20ef7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:20:22.268 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.265+0000 7f20ef7fe640 10 monclient: dump: 2024-03-25T19:20:22.268 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:20:22.268 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:20:22.268 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:22.268 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:22.268 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:22.268 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:20:22.268 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:20:22.268 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:20:22.268 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:20:22.268 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:20:22.268 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.265+0000 7f20f6b9d640 10 monclient: get_monmap_and_config success 2024-03-25T19:20:22.268 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.265+0000 7f20f6b9d640 4 set_mon_vals no callback set 2024-03-25T19:20:22.268 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.266+0000 7f20f6b9d640 10 monclient: shutdown 2024-03-25T19:20:22.268 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.266+0000 7f20f6b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:22.268 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.266+0000 7f20f6b9d640 1 -- 172.21.15.83:0/4253289674 >> v1:172.21.15.103:6789/0 conn(0x7f20f00b4d60 legacy=0x7f20f013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:22.268 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.266+0000 7f20f6b9d640 1 -- 172.21.15.83:0/4253289674 shutdown_connections 2024-03-25T19:20:22.269 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.266+0000 7f20f6b9d640 1 -- 172.21.15.83:0/4253289674 >> 172.21.15.83:0/4253289674 conn(0x7f20f00617a0 msgr2=0x7f20f0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:22.269 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.266+0000 7f20f6b9d640 1 -- 172.21.15.83:0/4253289674 shutdown_connections 2024-03-25T19:20:22.269 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.266+0000 7f20f6b9d640 1 -- 172.21.15.83:0/4253289674 wait complete. 2024-03-25T19:20:22.269 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.267+0000 7f20f6b9d640 10 monclient: build_initial_monmap 2024-03-25T19:20:22.269 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.267+0000 7f20f6b9d640 10 monclient: monmap: 2024-03-25T19:20:22.269 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:20:22.269 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:20:22.269 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:20:22.269 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:20:22.269 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:22.269 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:20:22.269 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:20:22.269 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:20:22.269 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:20:22.269 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:20:22.269 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.267+0000 7f20f6b9d640 1 librados: starting msgr at 2024-03-25T19:20:22.269 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.267+0000 7f20f6b9d640 1 librados: starting objecter 2024-03-25T19:20:22.270 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.267+0000 7f20f6b9d640 1 Processor -- start 2024-03-25T19:20:22.270 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.268+0000 7f20f6b9d640 1 -- start start 2024-03-25T19:20:22.270 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.268+0000 7f20f6b9d640 1 librados: setting wanted keys 2024-03-25T19:20:22.270 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.268+0000 7f20f6b9d640 1 librados: calling monclient init 2024-03-25T19:20:22.270 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.268+0000 7f20f6b9d640 10 monclient: init 2024-03-25T19:20:22.272 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.269+0000 7f20f6b9d640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:22.272 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.269+0000 7f20f6b9d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:20:22.272 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.269+0000 7f20f6b9d640 10 monclient(hunting): picked mon.noname-b con 0x7f20f00b1730 addr v1:172.21.15.103:6789/0 2024-03-25T19:20:22.272 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.269+0000 7f20f6b9d640 10 monclient(hunting): picked mon.noname-a con 0x7f20f00b4d60 addr v1:172.21.15.83:6789/0 2024-03-25T19:20:22.272 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.269+0000 7f20f6b9d640 10 monclient(hunting): picked mon.noname-c con 0x7f20f014be90 addr v1:172.21.15.83:6790/0 2024-03-25T19:20:22.272 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.269+0000 7f20f6b9d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20f0141c80 con 0x7f20f00b4d60 2024-03-25T19:20:22.272 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.269+0000 7f20f6b9d640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20f01403f0 con 0x7f20f00b1730 2024-03-25T19:20:22.273 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.269+0000 7f20f6b9d640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20f013dba0 con 0x7f20f014be90 2024-03-25T19:20:22.273 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.269+0000 7f20f6b9d640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:22.273 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.269+0000 7f20f6b9d640 10 monclient(hunting): authenticate will time out at 6079.541175s 2024-03-25T19:20:22.273 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.269+0000 7f20effff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f20f00b4d60 0x7f20f013d490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:48006/0 (socket says 172.21.15.83:48006) 2024-03-25T19:20:22.273 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.269+0000 7f20effff640 1 -- 172.21.15.83:0/954156996 learned_addr learned my addr 172.21.15.83:0/954156996 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:20:22.273 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.270+0000 7f20ed7fa640 10 client.?.objecter ms_handle_connect 0x7f20f00b4d60 2024-03-25T19:20:22.273 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.270+0000 7f20ed7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:22.273 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.270+0000 7f20ed7fa640 10 client.?.objecter ms_handle_connect 0x7f20f00b1730 2024-03-25T19:20:22.273 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.270+0000 7f20ed7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:22.273 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.270+0000 7f20ed7fa640 10 client.?.objecter ms_handle_connect 0x7f20f014be90 2024-03-25T19:20:22.273 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.270+0000 7f20ed7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:22.273 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.272+0000 7f20ed7fa640 1 -- 172.21.15.83:0/954156996 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3243584154 0 0) 0x7f20f0141c80 con 0x7f20f00b4d60 2024-03-25T19:20:22.273 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.272+0000 7f20ed7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:20:22.273 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.272+0000 7f20ed7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:22.273 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.272+0000 7f20ed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:22.273 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.272+0000 7f20ed7fa640 10 monclient(hunting): my global_id is 4221 2024-03-25T19:20:22.274 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.272+0000 7f20ed7fa640 1 -- 172.21.15.83:0/954156996 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20d4004eb0 con 0x7f20f00b4d60 2024-03-25T19:20:22.274 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.272+0000 7f20ed7fa640 1 -- 172.21.15.83:0/954156996 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3940185439 0 0) 0x7f20f01403f0 con 0x7f20f00b1730 2024-03-25T19:20:22.274 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.272+0000 7f20ed7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:20:22.274 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.272+0000 7f20ed7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:22.274 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.272+0000 7f20ed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:22.274 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.272+0000 7f20ed7fa640 10 monclient(hunting): my global_id is 4228 2024-03-25T19:20:22.274 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.272+0000 7f20ed7fa640 1 -- 172.21.15.83:0/954156996 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20f0141c80 con 0x7f20f00b1730 2024-03-25T19:20:22.275 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.273+0000 7f20ed7fa640 1 -- 172.21.15.83:0/954156996 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2030934760 0 0) 0x7f20f013dba0 con 0x7f20f014be90 2024-03-25T19:20:22.275 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.273+0000 7f20ed7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:20:22.275 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.273+0000 7f20ed7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:22.275 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.274+0000 7f20ed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:22.275 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.274+0000 7f20ed7fa640 10 monclient(hunting): my global_id is 4232 2024-03-25T19:20:22.275 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.274+0000 7f20ed7fa640 1 -- 172.21.15.83:0/954156996 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20f01403f0 con 0x7f20f014be90 2024-03-25T19:20:22.277 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.277+0000 7f20ed7fa640 1 -- 172.21.15.83:0/954156996 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4092769249 0 0) 0x7f20d4004eb0 con 0x7f20f00b4d60 2024-03-25T19:20:22.277 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.277+0000 7f20ed7fa640 1 -- 172.21.15.83:0/954156996 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f20f013dba0 con 0x7f20f00b4d60 2024-03-25T19:20:22.277 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.277+0000 7f20ed7fa640 1 -- 172.21.15.83:0/954156996 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f20dc001950 con 0x7f20f00b4d60 2024-03-25T19:20:22.277 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.277+0000 7f20ed7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:22.277 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.277+0000 7f20ed7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:20:22.277 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.277+0000 7f20ed7fa640 10 monclient(hunting): dump: 2024-03-25T19:20:22.277 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:20:22.277 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:20:22.278 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:22.278 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:22.278 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:22.278 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:20:22.278 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:20:22.278 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:20:22.278 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:20:22.278 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:20:22.278 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.277+0000 7f20ed7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:20:22.279 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.278+0000 7f20ed7fa640 1 -- 172.21.15.83:0/954156996 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3115100752 0 0) 0x7f20f0141c80 con 0x7f20f00b1730 2024-03-25T19:20:22.279 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.278+0000 7f20ed7fa640 1 -- 172.21.15.83:0/954156996 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f20d4004eb0 con 0x7f20f00b1730 2024-03-25T19:20:22.279 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.279+0000 7f20ed7fa640 1 -- 172.21.15.83:0/954156996 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f20e4001a60 con 0x7f20f00b1730 2024-03-25T19:20:22.279 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.279+0000 7f20ed7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:22.279 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.279+0000 7f20ed7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:20:22.279 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.279+0000 7f20ed7fa640 10 monclient(hunting): dump: 2024-03-25T19:20:22.279 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:20:22.279 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:20:22.279 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:22.279 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:22.279 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:22.279 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:20:22.280 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:20:22.280 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:20:22.280 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:20:22.280 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:20:22.282 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.280+0000 7f20ed7fa640 1 -- 172.21.15.83:0/954156996 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1948311370 0 0) 0x7f20f013dba0 con 0x7f20f00b4d60 2024-03-25T19:20:22.282 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.280+0000 7f20ed7fa640 1 -- 172.21.15.83:0/954156996 >> v1:172.21.15.83:6790/0 conn(0x7f20f014be90 legacy=0x7f20f013e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:22.282 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.280+0000 7f20ed7fa640 1 -- 172.21.15.83:0/954156996 >> v1:172.21.15.103:6789/0 conn(0x7f20f00b1730 legacy=0x7f20f013c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:22.282 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.280+0000 7f20ed7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:20:22.282 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.280+0000 7f20ed7fa640 1 monclient: found mon.a 2024-03-25T19:20:22.282 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.280+0000 7f20ed7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:22.283 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.280+0000 7f20ed7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:20:22.283 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.280+0000 7f20ed7fa640 1 -- 172.21.15.83:0/954156996 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20f013e7d0 con 0x7f20f00b4d60 2024-03-25T19:20:22.283 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.281+0000 7f20ed7fa640 10 monclient: _finish_auth 0 2024-03-25T19:20:22.283 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.281+0000 7f20ed7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:20:22.283 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.281+0000 7f20ed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:22.283 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.281+0000 7f20f6b9d640 5 monclient: authenticate success, global_id 4221 2024-03-25T19:20:22.283 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.281+0000 7f20f6b9d640 10 monclient: _renew_subs 2024-03-25T19:20:22.283 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.281+0000 7f20f6b9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:20:22.283 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.281+0000 7f20f6b9d640 1 -- 172.21.15.83:0/954156996 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20f013ea20 con 0x7f20f00b4d60 2024-03-25T19:20:22.283 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.281+0000 7f20f6b9d640 10 client.4221.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:20:22.283 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.281+0000 7f20f6b9d640 10 monclient: _renew_subs 2024-03-25T19:20:22.283 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.281+0000 7f20f6b9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:20:22.283 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.281+0000 7f20f6b9d640 1 -- 172.21.15.83:0/954156996 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f20f0141640 con 0x7f20f00b4d60 2024-03-25T19:20:22.283 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.281+0000 7f20f6b9d640 1 librados: init done 2024-03-25T19:20:22.283 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.282+0000 7f20f6b9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:22.283 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.282+0000 7f20f6b9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:22.283 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.282+0000 7f20f6b9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:20:22.284 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.282+0000 7f20f6b9d640 1 -- 172.21.15.83:0/954156996 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20f00627b0 con 0x7f20f00b4d60 2024-03-25T19:20:22.299 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.299+0000 7f20ed7fa640 1 -- 172.21.15.83:0/954156996 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20dc0047e0 con 0x7f20f00b4d60 2024-03-25T19:20:22.299 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.299+0000 7f20ed7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:22.299 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.299+0000 7f20ed7fa640 1 -- 172.21.15.83:0/954156996 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f20dc004af0 con 0x7f20f00b4d60 2024-03-25T19:20:22.299 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.299+0000 7f20ed7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:22.299 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.299+0000 7f20ed7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:20:22.299 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.299+0000 7f20ed7fa640 10 monclient: dump: 2024-03-25T19:20:22.299 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:20:22.299 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:20:22.299 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:22.299 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:22.299 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:22.299 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:20:22.299 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:20:22.300 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:20:22.300 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:20:22.300 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:20:22.300 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.299+0000 7f20eeffd640 4 set_mon_vals no callback set 2024-03-25T19:20:22.300 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.300+0000 7f20ed7fa640 1 -- 172.21.15.83:0/954156996 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7f20dc01aac0 con 0x7f20f00b4d60 2024-03-25T19:20:22.301 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.300+0000 7f20f4939640 10 monclient: get_auth_request con 0x7f20d406df70 auth_method 0 2024-03-25T19:20:22.301 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.301+0000 7f20ed7fa640 10 client.4221.objecter ms_handle_connect 0x7f20d406df70 2024-03-25T19:20:22.302 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.302+0000 7f20ed7fa640 1 -- 172.21.15.83:0/954156996 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7f20dc054860 con 0x7f20f00b4d60 2024-03-25T19:20:22.302 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.302+0000 7f20ed7fa640 10 client.4221.objecter ms_dispatch 0x7f20f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:20:22.302 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.302+0000 7f20ed7fa640 3 client.4221.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:20:22.303 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.302+0000 7f20ed7fa640 3 client.4221.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:20:22.303 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.302+0000 7f20ed7fa640 20 client.4221.objecter dump_active .. 0 homeless 2024-03-25T19:20:22.543 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.544+0000 7f20ed7fa640 1 -- 172.21.15.83:0/954156996 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f20dc0501a0 con 0x7f20f00b4d60 2024-03-25T19:20:22.543 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.544+0000 7f20ed7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:22.543 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.544+0000 7f20ed7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:20:22.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.729+0000 7f20f6b9d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:22.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.729+0000 7f20f6b9d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:22.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.729+0000 7f20f6b9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:20:22.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.729+0000 7f20f6b9d640 1 -- 172.21.15.83:0/954156996 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f20f013ebb0 con 0x7f20f00b4d60 2024-03-25T19:20:22.744 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.744+0000 7f20ed7fa640 1 -- 172.21.15.83:0/954156996 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7f20dc0522a0 con 0x7f20f00b4d60 2024-03-25T19:20:22.744 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.744+0000 7f20ed7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:22.744 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.744+0000 7f20ed7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:20:22.744 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:20:22.744 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:20:22.746 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.746+0000 7f20f6b9d640 10 librados: watch_flush enter 2024-03-25T19:20:22.747 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.747+0000 7f20f6b9d640 10 librados: watch_flush exit 2024-03-25T19:20:22.747 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.747+0000 7f20f6b9d640 20 client.4221.objecter shutdown clearing up homeless session... 2024-03-25T19:20:22.747 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.747+0000 7f20f6b9d640 10 client.4221.objecter successfully canceled tick 2024-03-25T19:20:22.747 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.747+0000 7f20f6b9d640 1 -- 172.21.15.83:0/954156996 >> v1:172.21.15.103:6800/345072327 conn(0x7f20d406df70 legacy=0x7f20d4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:22.747 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.747+0000 7f20f6b9d640 10 monclient: shutdown 2024-03-25T19:20:22.747 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.747+0000 7f20f6b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:22.747 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.747+0000 7f20f6b9d640 1 -- 172.21.15.83:0/954156996 >> v1:172.21.15.83:6789/0 conn(0x7f20f00b4d60 legacy=0x7f20f013d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:22.748 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.747+0000 7f20f6b9d640 1 -- 172.21.15.83:0/954156996 shutdown_connections 2024-03-25T19:20:22.748 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.748+0000 7f20f6b9d640 1 -- 172.21.15.83:0/954156996 >> 172.21.15.83:0/954156996 conn(0x7f20f00617a0 msgr2=0x7f20f0143900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:22.748 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.748+0000 7f20f6b9d640 1 -- 172.21.15.83:0/954156996 shutdown_connections 2024-03-25T19:20:22.748 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.748+0000 7f20f6b9d640 1 -- 172.21.15.83:0/954156996 wait complete. 2024-03-25T19:20:22.748 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:22.748+0000 7f20f6b9d640 1 librados: shutdown 2024-03-25T19:20:22.761 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:20:24.906 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-03-25T19:20:24.904+0000 5dc3480 -1 osd.5 0 log_to_monitors true 2024-03-25T19:20:25.458 INFO:tasks.ceph.osd.7.smithi103.stderr:2024-03-25T19:20:25.456+0000 5dc3480 -1 osd.7 0 log_to_monitors true 2024-03-25T19:20:25.638 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-03-25T19:20:25.635+0000 5dc3480 -1 osd.4 0 log_to_monitors true 2024-03-25T19:20:25.924 INFO:tasks.ceph.osd.6.smithi103.stderr:2024-03-25T19:20:25.875+0000 5dc3480 -1 osd.6 0 log_to_monitors true 2024-03-25T19:20:26.421 INFO:tasks.ceph.osd.1.smithi083.stderr:2024-03-25T19:20:26.418+0000 5dc3480 -1 osd.1 0 log_to_monitors true 2024-03-25T19:20:26.856 INFO:tasks.ceph.osd.2.smithi083.stderr:2024-03-25T19:20:26.853+0000 5dc3480 -1 osd.2 0 log_to_monitors true 2024-03-25T19:20:26.952 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-03-25T19:20:26.949+0000 5dc3480 -1 osd.3 0 log_to_monitors true 2024-03-25T19:20:27.100 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T19:20:27.099+0000 5dc3480 -1 osd.0 0 log_to_monitors true 2024-03-25T19:20:29.567 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:20:29.686 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.686+0000 7f62299bc640 10 monclient: get_monmap_and_config 2024-03-25T19:20:29.686 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.686+0000 7f62299bc640 10 monclient: build_initial_monmap 2024-03-25T19:20:29.686 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.686+0000 7f62299bc640 10 monclient: monmap: 2024-03-25T19:20:29.686 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:20:29.686 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:20:29.686 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:20:29.687089+0000 2024-03-25T19:20:29.686 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:20:29.687089+0000 2024-03-25T19:20:29.686 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:29.686 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:20:29.686 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:20:29.686 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:20:29.686 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:20:29.687 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:20:29.688 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.687+0000 7f62299bc640 1 Processor -- start 2024-03-25T19:20:29.688 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.687+0000 7f62299bc640 1 -- start start 2024-03-25T19:20:29.688 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.687+0000 7f62299bc640 10 monclient: init 2024-03-25T19:20:29.688 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.687+0000 7f62299bc640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:29.688 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.687+0000 7f62299bc640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:20:29.688 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.688+0000 7f62299bc640 10 monclient(hunting): picked mon.noname-a con 0x7f62240a98f0 addr v1:172.21.15.83:6789/0 2024-03-25T19:20:29.688 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.688+0000 7f62299bc640 10 monclient(hunting): picked mon.noname-b con 0x7f622413f710 addr v1:172.21.15.83:6790/0 2024-03-25T19:20:29.689 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.688+0000 7f62299bc640 10 monclient(hunting): picked mon.noname-c con 0x7f62241430c0 addr v1:172.21.15.103:6789/0 2024-03-25T19:20:29.689 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.688+0000 7f62299bc640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6224142520 con 0x7f62240a98f0 2024-03-25T19:20:29.689 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.688+0000 7f62299bc640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62240b28a0 con 0x7f62241430c0 2024-03-25T19:20:29.689 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.688+0000 7f62299bc640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62240b2a30 con 0x7f622413f710 2024-03-25T19:20:29.689 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.688+0000 7f62299bc640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:29.689 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.688+0000 7f62299bc640 10 monclient(hunting): authenticate will time out at 6086.960081s 2024-03-25T19:20:29.700 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.700+0000 7f62227fc640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f622413f710 0x7f62241429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:55722/0 (socket says 172.21.15.83:55722) 2024-03-25T19:20:29.700 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.700+0000 7f62227fc640 1 -- 172.21.15.83:0/33335166 learned_addr learned my addr 172.21.15.83:0/33335166 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:20:29.714 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.715+0000 7f6221ffb640 1 -- 172.21.15.83:0/33335166 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 230182841 0 0) 0x7f62240b2a30 con 0x7f622413f710 2024-03-25T19:20:29.714 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.715+0000 7f6221ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:20:29.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.715+0000 7f6221ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:29.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.715+0000 7f6221ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:29.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.715+0000 7f6221ffb640 10 monclient(hunting): my global_id is 4259 2024-03-25T19:20:29.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.715+0000 7f6221ffb640 1 -- 172.21.15.83:0/33335166 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f620c003520 con 0x7f622413f710 2024-03-25T19:20:29.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.716+0000 7f6221ffb640 1 -- 172.21.15.83:0/33335166 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3883114143 0 0) 0x7f62240b28a0 con 0x7f62241430c0 2024-03-25T19:20:29.716 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.716+0000 7f6221ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:20:29.716 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.716+0000 7f6221ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:29.716 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.716+0000 7f6221ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:29.716 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.716+0000 7f6221ffb640 10 monclient(hunting): my global_id is 4255 2024-03-25T19:20:29.716 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.716+0000 7f6221ffb640 1 -- 172.21.15.83:0/33335166 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62240b2a30 con 0x7f62241430c0 2024-03-25T19:20:29.719 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.719+0000 7f6221ffb640 1 -- 172.21.15.83:0/33335166 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 779454986 0 0) 0x7f6224142520 con 0x7f62240a98f0 2024-03-25T19:20:29.719 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.719+0000 7f6221ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:20:29.719 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.719+0000 7f6221ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:29.720 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.719+0000 7f6221ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:29.720 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.719+0000 7f6221ffb640 10 monclient(hunting): my global_id is 4248 2024-03-25T19:20:29.720 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.719+0000 7f6221ffb640 1 -- 172.21.15.83:0/33335166 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62240b28a0 con 0x7f62240a98f0 2024-03-25T19:20:29.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.728+0000 7f6221ffb640 1 -- 172.21.15.83:0/33335166 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1412159038 0 0) 0x7f62240b2a30 con 0x7f62241430c0 2024-03-25T19:20:29.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.728+0000 7f6221ffb640 1 -- 172.21.15.83:0/33335166 >> v1:172.21.15.83:6790/0 conn(0x7f622413f710 legacy=0x7f62241429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:29.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.728+0000 7f6221ffb640 1 -- 172.21.15.83:0/33335166 >> v1:172.21.15.83:6789/0 conn(0x7f62240a98f0 legacy=0x7f62240a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:29.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.728+0000 7f6221ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:20:29.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.728+0000 7f6221ffb640 1 monclient: found mon.noname-c 2024-03-25T19:20:29.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.728+0000 7f6221ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:29.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.728+0000 7f6221ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.103:6789/0 2024-03-25T19:20:29.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.728+0000 7f6221ffb640 1 -- 172.21.15.83:0/33335166 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62240b44c0 con 0x7f62241430c0 2024-03-25T19:20:29.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.728+0000 7f6221ffb640 10 monclient: _finish_auth 0 2024-03-25T19:20:29.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.728+0000 7f6221ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:20:29.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.728+0000 7f6221ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:29.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.728+0000 7f6221ffb640 1 -- 172.21.15.83:0/33335166 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f621c0019f0 con 0x7f62241430c0 2024-03-25T19:20:29.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.728+0000 7f6221ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:29.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.728+0000 7f6221ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:20:29.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.728+0000 7f6221ffb640 10 monclient: dump: 2024-03-25T19:20:29.729 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:20:29.729 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:20:29.729 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:29.729 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:29.729 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:29.729 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:20:29.730 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:20:29.730 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:20:29.730 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:20:29.730 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:20:29.730 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.728+0000 7f62299bc640 5 monclient: authenticate success, global_id 4255 2024-03-25T19:20:29.730 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.728+0000 7f62299bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:20:29.744 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.745+0000 7f6221ffb640 1 -- 172.21.15.83:0/33335166 <== mon.1 v1:172.21.15.103:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f621c001b50 con 0x7f62241430c0 2024-03-25T19:20:29.745 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.745+0000 7f6221ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:29.745 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.745+0000 7f6221ffb640 1 -- 172.21.15.83:0/33335166 <== mon.1 v1:172.21.15.103:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f621c001f60 con 0x7f62241430c0 2024-03-25T19:20:29.745 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.745+0000 7f6221ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:29.745 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.745+0000 7f6221ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:20:29.745 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.745+0000 7f6221ffb640 10 monclient: dump: 2024-03-25T19:20:29.745 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:20:29.745 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:20:29.745 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:29.745 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:29.745 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:29.745 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:20:29.745 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:20:29.745 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:20:29.746 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:20:29.746 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:20:29.746 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.745+0000 7f62299bc640 10 monclient: get_monmap_and_config success 2024-03-25T19:20:29.746 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.745+0000 7f62299bc640 4 set_mon_vals no callback set 2024-03-25T19:20:29.748 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.745+0000 7f62299bc640 10 monclient: shutdown 2024-03-25T19:20:29.748 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.745+0000 7f62299bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:29.748 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.745+0000 7f62299bc640 1 -- 172.21.15.83:0/33335166 >> v1:172.21.15.103:6789/0 conn(0x7f62241430c0 legacy=0x7f622414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:29.748 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.746+0000 7f62299bc640 1 -- 172.21.15.83:0/33335166 shutdown_connections 2024-03-25T19:20:29.748 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.746+0000 7f62299bc640 1 -- 172.21.15.83:0/33335166 >> 172.21.15.83:0/33335166 conn(0x7f62240b7ea0 msgr2=0x7f62240b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:29.748 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.746+0000 7f62299bc640 1 -- 172.21.15.83:0/33335166 shutdown_connections 2024-03-25T19:20:29.748 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.746+0000 7f62299bc640 1 -- 172.21.15.83:0/33335166 wait complete. 2024-03-25T19:20:29.748 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.747+0000 7f62299bc640 10 monclient: build_initial_monmap 2024-03-25T19:20:29.748 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.747+0000 7f62299bc640 10 monclient: monmap: 2024-03-25T19:20:29.748 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:20:29.749 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:20:29.749 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:20:29.749 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:20:29.749 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:29.749 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:20:29.749 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:20:29.749 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:20:29.749 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:20:29.749 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:20:29.749 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.747+0000 7f62299bc640 1 librados: starting msgr at 2024-03-25T19:20:29.749 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.747+0000 7f62299bc640 1 librados: starting objecter 2024-03-25T19:20:29.749 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.747+0000 7f62299bc640 1 Processor -- start 2024-03-25T19:20:29.749 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.747+0000 7f62299bc640 1 -- start start 2024-03-25T19:20:29.749 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.747+0000 7f62299bc640 1 librados: setting wanted keys 2024-03-25T19:20:29.750 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.747+0000 7f62299bc640 1 librados: calling monclient init 2024-03-25T19:20:29.750 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.747+0000 7f62299bc640 10 monclient: init 2024-03-25T19:20:29.750 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.748+0000 7f62299bc640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:29.750 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.748+0000 7f62299bc640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:20:29.750 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.748+0000 7f62299bc640 10 monclient(hunting): picked mon.noname-a con 0x7f62240a98f0 addr v1:172.21.15.83:6789/0 2024-03-25T19:20:29.750 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.748+0000 7f62299bc640 10 monclient(hunting): picked mon.noname-c con 0x7f622413f710 addr v1:172.21.15.83:6790/0 2024-03-25T19:20:29.750 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.748+0000 7f62299bc640 10 monclient(hunting): picked mon.noname-b con 0x7f62241430c0 addr v1:172.21.15.103:6789/0 2024-03-25T19:20:29.750 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.748+0000 7f62299bc640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6224133ee0 con 0x7f62240a98f0 2024-03-25T19:20:29.750 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.748+0000 7f62299bc640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62240abe30 con 0x7f62241430c0 2024-03-25T19:20:29.750 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.748+0000 7f62299bc640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62240ac0a0 con 0x7f622413f710 2024-03-25T19:20:29.750 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.748+0000 7f62299bc640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:29.750 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.748+0000 7f62299bc640 10 monclient(hunting): authenticate will time out at 6087.020257s 2024-03-25T19:20:29.759 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.760+0000 7f62237fe640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f62241430c0 0x7f6224138dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:36946/0 (socket says 172.21.15.83:36946) 2024-03-25T19:20:29.760 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.760+0000 7f62237fe640 1 -- 172.21.15.83:0/2817977409 learned_addr learned my addr 172.21.15.83:0/2817977409 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:20:29.763 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.764+0000 7f620bfff640 10 client.?.objecter ms_handle_connect 0x7f62241430c0 2024-03-25T19:20:29.764 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.764+0000 7f620bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:29.766 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.766+0000 7f620bfff640 10 client.?.objecter ms_handle_connect 0x7f62240a98f0 2024-03-25T19:20:29.766 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.766+0000 7f620bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:29.772 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.772+0000 7f620bfff640 10 client.?.objecter ms_handle_connect 0x7f622413f710 2024-03-25T19:20:29.772 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.772+0000 7f620bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:29.778 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.778+0000 7f620bfff640 1 -- 172.21.15.83:0/2817977409 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1333353845 0 0) 0x7f62240ac0a0 con 0x7f622413f710 2024-03-25T19:20:29.778 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.778+0000 7f620bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:20:29.779 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.778+0000 7f620bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:29.779 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.778+0000 7f620bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:29.779 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.778+0000 7f620bfff640 10 monclient(hunting): my global_id is 4262 2024-03-25T19:20:29.779 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.778+0000 7f620bfff640 1 -- 172.21.15.83:0/2817977409 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6210004eb0 con 0x7f622413f710 2024-03-25T19:20:29.784 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.784+0000 7f620bfff640 1 -- 172.21.15.83:0/2817977409 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3444468035 0 0) 0x7f62240b28a0 con 0x7f62240a98f0 2024-03-25T19:20:29.784 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.784+0000 7f620bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:20:29.784 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.784+0000 7f620bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:29.784 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.784+0000 7f620bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:29.784 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.784+0000 7f620bfff640 10 monclient(hunting): my global_id is 4251 2024-03-25T19:20:29.784 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.784+0000 7f620bfff640 1 -- 172.21.15.83:0/2817977409 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62240ac0a0 con 0x7f62240a98f0 2024-03-25T19:20:29.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.787+0000 7f620bfff640 1 -- 172.21.15.83:0/2817977409 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4140140914 0 0) 0x7f62240abe30 con 0x7f62241430c0 2024-03-25T19:20:29.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.787+0000 7f620bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:20:29.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.787+0000 7f620bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:29.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.787+0000 7f620bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:29.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.787+0000 7f620bfff640 10 monclient(hunting): my global_id is 4258 2024-03-25T19:20:29.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.787+0000 7f620bfff640 1 -- 172.21.15.83:0/2817977409 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62240b28a0 con 0x7f62241430c0 2024-03-25T19:20:29.789 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.790+0000 7f620bfff640 1 -- 172.21.15.83:0/2817977409 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4253587043 0 0) 0x7f6210004eb0 con 0x7f622413f710 2024-03-25T19:20:29.789 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.790+0000 7f620bfff640 1 -- 172.21.15.83:0/2817977409 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62240abe30 con 0x7f622413f710 2024-03-25T19:20:29.791 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.792+0000 7f620bfff640 1 -- 172.21.15.83:0/2817977409 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f6218001910 con 0x7f622413f710 2024-03-25T19:20:29.792 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.792+0000 7f620bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:29.792 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.792+0000 7f620bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:20:29.792 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.792+0000 7f620bfff640 10 monclient(hunting): dump: 2024-03-25T19:20:29.792 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:20:29.792 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:20:29.792 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:29.792 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:29.792 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:29.792 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:20:29.792 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:20:29.792 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:20:29.792 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:20:29.792 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:20:29.792 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.792+0000 7f620bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:20:29.799 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.799+0000 7f620bfff640 1 -- 172.21.15.83:0/2817977409 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3553021511 0 0) 0x7f62240b28a0 con 0x7f62241430c0 2024-03-25T19:20:29.799 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.800+0000 7f620bfff640 1 -- 172.21.15.83:0/2817977409 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6210004eb0 con 0x7f62241430c0 2024-03-25T19:20:29.799 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.800+0000 7f620bfff640 1 -- 172.21.15.83:0/2817977409 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f621c001f20 con 0x7f62241430c0 2024-03-25T19:20:29.799 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.800+0000 7f620bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:29.800 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.800+0000 7f620bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:20:29.800 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.800+0000 7f620bfff640 10 monclient(hunting): dump: 2024-03-25T19:20:29.800 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:20:29.800 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:20:29.800 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:29.800 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:29.800 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:29.800 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:20:29.800 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:20:29.800 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:20:29.800 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:20:29.800 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:20:29.801 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.800+0000 7f620bfff640 1 -- 172.21.15.83:0/2817977409 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 956450028 0 0) 0x7f62240ac0a0 con 0x7f62240a98f0 2024-03-25T19:20:29.802 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.800+0000 7f620bfff640 1 -- 172.21.15.83:0/2817977409 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62240b28a0 con 0x7f62240a98f0 2024-03-25T19:20:29.802 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.800+0000 7f620bfff640 1 -- 172.21.15.83:0/2817977409 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f62140016d0 con 0x7f62240a98f0 2024-03-25T19:20:29.802 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.800+0000 7f620bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:29.802 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.800+0000 7f620bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:20:29.802 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.800+0000 7f620bfff640 10 monclient(hunting): dump: 2024-03-25T19:20:29.802 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:20:29.802 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:20:29.802 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:29.802 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:29.802 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:29.802 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:20:29.802 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:20:29.802 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:20:29.803 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:20:29.803 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:20:29.803 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.801+0000 7f620bfff640 1 -- 172.21.15.83:0/2817977409 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3358487567 0 0) 0x7f62240abe30 con 0x7f622413f710 2024-03-25T19:20:29.803 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.801+0000 7f620bfff640 1 -- 172.21.15.83:0/2817977409 >> v1:172.21.15.103:6789/0 conn(0x7f62241430c0 legacy=0x7f6224138dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:29.803 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.801+0000 7f620bfff640 1 -- 172.21.15.83:0/2817977409 >> v1:172.21.15.83:6789/0 conn(0x7f62240a98f0 legacy=0x7f62241367f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:29.803 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.802+0000 7f620bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:20:29.803 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.802+0000 7f620bfff640 1 monclient: found mon.c 2024-03-25T19:20:29.803 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.802+0000 7f620bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:29.803 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.802+0000 7f620bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:20:29.803 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.802+0000 7f620bfff640 1 -- 172.21.15.83:0/2817977409 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62240ac380 con 0x7f622413f710 2024-03-25T19:20:29.803 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.802+0000 7f620bfff640 10 monclient: _finish_auth 0 2024-03-25T19:20:29.803 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.802+0000 7f620bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:20:29.803 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.802+0000 7f620bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:29.803 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.802+0000 7f62299bc640 5 monclient: authenticate success, global_id 4262 2024-03-25T19:20:29.803 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.802+0000 7f62299bc640 10 monclient: _renew_subs 2024-03-25T19:20:29.803 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.802+0000 7f62299bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:20:29.804 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.802+0000 7f62299bc640 1 -- 172.21.15.83:0/2817977409 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62241381c0 con 0x7f622413f710 2024-03-25T19:20:29.804 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.802+0000 7f62299bc640 10 client.4262.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:20:29.804 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.802+0000 7f62299bc640 10 monclient: _renew_subs 2024-03-25T19:20:29.804 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.802+0000 7f62299bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:20:29.804 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.802+0000 7f62299bc640 1 -- 172.21.15.83:0/2817977409 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f62241e36c0 con 0x7f622413f710 2024-03-25T19:20:29.804 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.802+0000 7f62299bc640 1 librados: init done 2024-03-25T19:20:29.804 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.803+0000 7f62299bc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:29.804 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.803+0000 7f62299bc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:29.804 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.803+0000 7f62299bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:20:29.805 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:29.803+0000 7f62299bc640 1 -- 172.21.15.83:0/2817977409 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62240b3ef0 con 0x7f622413f710 2024-03-25T19:20:30.097 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:30.098+0000 7f620bfff640 1 -- 172.21.15.83:0/2817977409 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62180047e0 con 0x7f622413f710 2024-03-25T19:20:30.098 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:30.098+0000 7f620bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:30.098 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:30.098+0000 7f620bfff640 1 -- 172.21.15.83:0/2817977409 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f6218004af0 con 0x7f622413f710 2024-03-25T19:20:30.098 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:30.098+0000 7f620bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:30.098 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:30.098+0000 7f620bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:20:30.098 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:30.098+0000 7f620bfff640 10 monclient: dump: 2024-03-25T19:20:30.098 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:20:30.098 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:20:30.098 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:30.098 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:30.098 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:30.098 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:20:30.098 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:20:30.098 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:20:30.098 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:20:30.098 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:20:30.099 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:30.098+0000 7f62217fa640 4 set_mon_vals no callback set 2024-03-25T19:20:30.099 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:30.098+0000 7f620bfff640 1 -- 172.21.15.83:0/2817977409 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7f621801aac0 con 0x7f622413f710 2024-03-25T19:20:30.099 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:30.099+0000 7f6222ffd640 10 monclient: get_auth_request con 0x7f621006e070 auth_method 0 2024-03-25T19:20:30.099 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:30.099+0000 7f620bfff640 10 client.4262.objecter ms_handle_connect 0x7f621006e070 2024-03-25T19:20:30.105 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:30.106+0000 7f620bfff640 1 -- 172.21.15.83:0/2817977409 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7f6218054840 con 0x7f622413f710 2024-03-25T19:20:30.105 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:30.106+0000 7f620bfff640 10 client.4262.objecter ms_dispatch 0x7f6224000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:20:30.106 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:30.106+0000 7f620bfff640 3 client.4262.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:20:30.106 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:30.106+0000 7f620bfff640 3 client.4262.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:20:30.106 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:30.106+0000 7f620bfff640 20 client.4262.objecter dump_active .. 0 homeless 2024-03-25T19:20:30.106 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:30.106+0000 7f620bfff640 1 -- 172.21.15.83:0/2817977409 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6218050180 con 0x7f622413f710 2024-03-25T19:20:30.106 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:30.106+0000 7f620bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:30.106 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:30.106+0000 7f620bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:20:30.329 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:30.329+0000 7f62299bc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:30.329 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:30.329+0000 7f62299bc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:30.329 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:30.329+0000 7f62299bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:20:30.329 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:30.329+0000 7f62299bc640 1 -- 172.21.15.83:0/2817977409 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6224138350 con 0x7f622413f710 2024-03-25T19:20:30.357 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:30.357+0000 7f620bfff640 1 -- 172.21.15.83:0/2817977409 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7f6218052280 con 0x7f622413f710 2024-03-25T19:20:30.357 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:30.357+0000 7f620bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:30.357 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:30.357+0000 7f620bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:20:30.357 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:20:30.357 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:20:30.360 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:30.359+0000 7f62299bc640 10 librados: watch_flush enter 2024-03-25T19:20:30.360 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:30.359+0000 7f62299bc640 10 librados: watch_flush exit 2024-03-25T19:20:30.360 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:30.359+0000 7f62299bc640 20 client.4262.objecter shutdown clearing up homeless session... 2024-03-25T19:20:30.360 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:30.359+0000 7f62299bc640 10 client.4262.objecter successfully canceled tick 2024-03-25T19:20:30.360 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:30.359+0000 7f62299bc640 1 -- 172.21.15.83:0/2817977409 >> v1:172.21.15.103:6800/345072327 conn(0x7f621006e070 legacy=0x7f6210070530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:30.360 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:30.359+0000 7f62299bc640 10 monclient: shutdown 2024-03-25T19:20:30.360 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:30.359+0000 7f62299bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:30.360 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:30.359+0000 7f62299bc640 1 -- 172.21.15.83:0/2817977409 >> v1:172.21.15.83:6790/0 conn(0x7f622413f710 legacy=0x7f62240ac7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:30.360 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:30.359+0000 7f62299bc640 1 -- 172.21.15.83:0/2817977409 shutdown_connections 2024-03-25T19:20:30.360 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:30.359+0000 7f62299bc640 1 -- 172.21.15.83:0/2817977409 >> 172.21.15.83:0/2817977409 conn(0x7f62240b7ea0 msgr2=0x7f6224074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:30.360 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:30.360+0000 7f62299bc640 1 -- 172.21.15.83:0/2817977409 shutdown_connections 2024-03-25T19:20:30.360 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:30.360+0000 7f62299bc640 1 -- 172.21.15.83:0/2817977409 wait complete. 2024-03-25T19:20:30.360 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:30.360+0000 7f62299bc640 1 librados: shutdown 2024-03-25T19:20:30.372 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:20:37.178 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:20:37.290 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.290+0000 7f036d896640 10 monclient: get_monmap_and_config 2024-03-25T19:20:37.290 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.290+0000 7f036d896640 10 monclient: build_initial_monmap 2024-03-25T19:20:37.290 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.290+0000 7f036d896640 10 monclient: monmap: 2024-03-25T19:20:37.290 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:20:37.290 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:20:37.290 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:20:37.291602+0000 2024-03-25T19:20:37.290 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:20:37.291602+0000 2024-03-25T19:20:37.290 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:37.291 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:20:37.291 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:20:37.291 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:20:37.291 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:20:37.291 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:20:37.292 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.291+0000 7f036d896640 1 Processor -- start 2024-03-25T19:20:37.292 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.291+0000 7f036d896640 1 -- start start 2024-03-25T19:20:37.292 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.291+0000 7f036d896640 10 monclient: init 2024-03-25T19:20:37.292 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.292+0000 7f036d896640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:37.292 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.292+0000 7f036d896640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:20:37.292 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.292+0000 7f036d896640 10 monclient(hunting): picked mon.noname-c con 0x7f03680a98f0 addr v1:172.21.15.103:6789/0 2024-03-25T19:20:37.292 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.292+0000 7f036d896640 10 monclient(hunting): picked mon.noname-a con 0x7f036813f710 addr v1:172.21.15.83:6789/0 2024-03-25T19:20:37.292 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.292+0000 7f036d896640 10 monclient(hunting): picked mon.noname-b con 0x7f03681430c0 addr v1:172.21.15.83:6790/0 2024-03-25T19:20:37.292 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.292+0000 7f036d896640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0368142520 con 0x7f036813f710 2024-03-25T19:20:37.293 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.292+0000 7f036d896640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03680b28a0 con 0x7f03680a98f0 2024-03-25T19:20:37.293 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.292+0000 7f036d896640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03680b2a30 con 0x7f03681430c0 2024-03-25T19:20:37.293 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.292+0000 7f036d896640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:37.293 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.292+0000 7f036d896640 10 monclient(hunting): authenticate will time out at 6094.564376s 2024-03-25T19:20:37.301 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.301+0000 7f03677fe640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f03681430c0 0x7f036814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:34816/0 (socket says 172.21.15.83:34816) 2024-03-25T19:20:37.301 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.301+0000 7f03677fe640 1 -- 172.21.15.83:0/3966251750 learned_addr learned my addr 172.21.15.83:0/3966251750 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:20:37.311 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.311+0000 7f0365ffb640 1 -- 172.21.15.83:0/3966251750 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2545556752 0 0) 0x7f0368142520 con 0x7f036813f710 2024-03-25T19:20:37.311 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.311+0000 7f0365ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:20:37.311 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.311+0000 7f0365ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:37.311 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.311+0000 7f0365ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:37.311 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.311+0000 7f0365ffb640 10 monclient(hunting): my global_id is 4254 2024-03-25T19:20:37.311 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.311+0000 7f0365ffb640 1 -- 172.21.15.83:0/3966251750 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0350003520 con 0x7f036813f710 2024-03-25T19:20:37.314 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.315+0000 7f0365ffb640 1 -- 172.21.15.83:0/3966251750 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1461096600 0 0) 0x7f03680b2a30 con 0x7f03681430c0 2024-03-25T19:20:37.314 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.315+0000 7f0365ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:20:37.315 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.315+0000 7f0365ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:37.315 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.315+0000 7f0365ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:37.315 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.315+0000 7f0365ffb640 10 monclient(hunting): my global_id is 4265 2024-03-25T19:20:37.315 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.315+0000 7f0365ffb640 1 -- 172.21.15.83:0/3966251750 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0368142520 con 0x7f03681430c0 2024-03-25T19:20:37.321 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.322+0000 7f0365ffb640 1 -- 172.21.15.83:0/3966251750 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1041958231 0 0) 0x7f0350003520 con 0x7f036813f710 2024-03-25T19:20:37.322 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.322+0000 7f0365ffb640 1 -- 172.21.15.83:0/3966251750 >> v1:172.21.15.83:6790/0 conn(0x7f03681430c0 legacy=0x7f036814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:37.322 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.322+0000 7f0365ffb640 1 -- 172.21.15.83:0/3966251750 >> v1:172.21.15.103:6789/0 conn(0x7f03680a98f0 legacy=0x7f03680a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:37.322 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.322+0000 7f0365ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:20:37.322 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.322+0000 7f0365ffb640 1 monclient: found mon.noname-a 2024-03-25T19:20:37.322 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.322+0000 7f0365ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:37.322 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.322+0000 7f0365ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.83:6789/0 2024-03-25T19:20:37.322 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.322+0000 7f0365ffb640 1 -- 172.21.15.83:0/3966251750 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03680b44c0 con 0x7f036813f710 2024-03-25T19:20:37.322 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.322+0000 7f0365ffb640 10 monclient: _finish_auth 0 2024-03-25T19:20:37.322 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.322+0000 7f0365ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:20:37.322 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.322+0000 7f0365ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:37.322 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.322+0000 7f0365ffb640 1 -- 172.21.15.83:0/3966251750 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f035c001690 con 0x7f036813f710 2024-03-25T19:20:37.322 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.322+0000 7f036d896640 5 monclient: authenticate success, global_id 4254 2024-03-25T19:20:37.322 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.322+0000 7f036d896640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:20:37.322 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.322+0000 7f0365ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:37.322 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.322+0000 7f0365ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:20:37.323 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.322+0000 7f0365ffb640 10 monclient: dump: 2024-03-25T19:20:37.323 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:20:37.323 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:20:37.323 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:37.323 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:37.323 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:37.323 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:20:37.323 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:20:37.323 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:20:37.323 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:20:37.323 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:20:37.323 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.322+0000 7f036d896640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:20:37.327 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.327+0000 7f0365ffb640 1 -- 172.21.15.83:0/3966251750 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f035c0017f0 con 0x7f036813f710 2024-03-25T19:20:37.327 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.327+0000 7f0365ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:37.327 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.327+0000 7f0365ffb640 1 -- 172.21.15.83:0/3966251750 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f035c001c00 con 0x7f036813f710 2024-03-25T19:20:37.327 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.327+0000 7f036d896640 10 monclient: get_monmap_and_config success 2024-03-25T19:20:37.327 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.327+0000 7f036d896640 4 set_mon_vals no callback set 2024-03-25T19:20:37.327 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.327+0000 7f0365ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:37.327 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.327+0000 7f0365ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:20:37.327 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.327+0000 7f0365ffb640 10 monclient: dump: 2024-03-25T19:20:37.327 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:20:37.327 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:20:37.327 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:37.327 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:37.327 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:37.327 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:20:37.328 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:20:37.328 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:20:37.328 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:20:37.328 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:20:37.329 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.328+0000 7f036d896640 10 monclient: shutdown 2024-03-25T19:20:37.329 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.328+0000 7f036d896640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:37.329 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.328+0000 7f036d896640 1 -- 172.21.15.83:0/3966251750 >> v1:172.21.15.83:6789/0 conn(0x7f036813f710 legacy=0x7f03681429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:37.329 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.328+0000 7f036d896640 1 -- 172.21.15.83:0/3966251750 shutdown_connections 2024-03-25T19:20:37.329 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.328+0000 7f036d896640 1 -- 172.21.15.83:0/3966251750 >> 172.21.15.83:0/3966251750 conn(0x7f03680b7ea0 msgr2=0x7f03680b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:37.329 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.328+0000 7f036d896640 1 -- 172.21.15.83:0/3966251750 shutdown_connections 2024-03-25T19:20:37.329 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.328+0000 7f036d896640 1 -- 172.21.15.83:0/3966251750 wait complete. 2024-03-25T19:20:37.329 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.329+0000 7f036d896640 10 monclient: build_initial_monmap 2024-03-25T19:20:37.329 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.329+0000 7f036d896640 10 monclient: monmap: 2024-03-25T19:20:37.329 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:20:37.330 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:20:37.330 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:20:37.330 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:20:37.330 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:37.330 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:20:37.330 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:20:37.330 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:20:37.330 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:20:37.330 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:20:37.330 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.329+0000 7f036d896640 1 librados: starting msgr at 2024-03-25T19:20:37.330 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.329+0000 7f036d896640 1 librados: starting objecter 2024-03-25T19:20:37.330 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.329+0000 7f036d896640 1 Processor -- start 2024-03-25T19:20:37.330 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.329+0000 7f036d896640 1 -- start start 2024-03-25T19:20:37.330 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.329+0000 7f036d896640 1 librados: setting wanted keys 2024-03-25T19:20:37.330 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.329+0000 7f036d896640 1 librados: calling monclient init 2024-03-25T19:20:37.330 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.329+0000 7f036d896640 10 monclient: init 2024-03-25T19:20:37.331 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.330+0000 7f036d896640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:37.331 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.330+0000 7f036d896640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:20:37.331 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.330+0000 7f036d896640 10 monclient(hunting): picked mon.noname-c con 0x7f03680a98f0 addr v1:172.21.15.83:6790/0 2024-03-25T19:20:37.331 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.330+0000 7f036d896640 10 monclient(hunting): picked mon.noname-b con 0x7f036813f710 addr v1:172.21.15.103:6789/0 2024-03-25T19:20:37.332 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.330+0000 7f036d896640 10 monclient(hunting): picked mon.noname-a con 0x7f03681430c0 addr v1:172.21.15.83:6789/0 2024-03-25T19:20:37.332 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.330+0000 7f036d896640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03681397c0 con 0x7f03681430c0 2024-03-25T19:20:37.332 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.330+0000 7f036d896640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03681daa20 con 0x7f036813f710 2024-03-25T19:20:37.332 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.330+0000 7f036d896640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03681d8210 con 0x7f03680a98f0 2024-03-25T19:20:37.332 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.330+0000 7f036d896640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:37.332 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.330+0000 7f036d896640 10 monclient(hunting): authenticate will time out at 6094.602322s 2024-03-25T19:20:37.341 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.341+0000 7f03667fc640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f036813f710 0x7f03681d73f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:37948/0 (socket says 172.21.15.83:37948) 2024-03-25T19:20:37.341 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.341+0000 7f03667fc640 1 -- 172.21.15.83:0/281931422 learned_addr learned my addr 172.21.15.83:0/281931422 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:20:37.342 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.343+0000 7f034ffff640 10 client.?.objecter ms_handle_connect 0x7f036813f710 2024-03-25T19:20:37.343 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.343+0000 7f034ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:37.344 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.344+0000 7f034ffff640 10 client.?.objecter ms_handle_connect 0x7f03680a98f0 2024-03-25T19:20:37.344 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.344+0000 7f034ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:37.354 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.354+0000 7f034ffff640 10 client.?.objecter ms_handle_connect 0x7f03681430c0 2024-03-25T19:20:37.354 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.354+0000 7f034ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:37.355 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.355+0000 7f034ffff640 1 -- 172.21.15.83:0/281931422 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 983729147 0 0) 0x7f03681daa20 con 0x7f036813f710 2024-03-25T19:20:37.355 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.355+0000 7f034ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:20:37.355 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.355+0000 7f034ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:37.355 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.355+0000 7f034ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:37.355 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.355+0000 7f034ffff640 10 monclient(hunting): my global_id is 4264 2024-03-25T19:20:37.355 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.355+0000 7f034ffff640 1 -- 172.21.15.83:0/281931422 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0354004eb0 con 0x7f036813f710 2024-03-25T19:20:37.358 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.358+0000 7f034ffff640 1 -- 172.21.15.83:0/281931422 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2009091985 0 0) 0x7f03681d8210 con 0x7f03680a98f0 2024-03-25T19:20:37.358 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.358+0000 7f034ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:20:37.358 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.358+0000 7f034ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:37.358 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.358+0000 7f034ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:37.358 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.358+0000 7f034ffff640 10 monclient(hunting): my global_id is 4268 2024-03-25T19:20:37.358 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.358+0000 7f034ffff640 1 -- 172.21.15.83:0/281931422 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03681daa20 con 0x7f03680a98f0 2024-03-25T19:20:37.368 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.369+0000 7f034ffff640 1 -- 172.21.15.83:0/281931422 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3966722252 0 0) 0x7f03681397c0 con 0x7f03681430c0 2024-03-25T19:20:37.368 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.369+0000 7f034ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:20:37.368 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.369+0000 7f034ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:37.368 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.369+0000 7f034ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:37.368 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.369+0000 7f034ffff640 10 monclient(hunting): my global_id is 4257 2024-03-25T19:20:37.368 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.369+0000 7f034ffff640 1 -- 172.21.15.83:0/281931422 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03681d8210 con 0x7f03681430c0 2024-03-25T19:20:37.370 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.370+0000 7f034ffff640 1 -- 172.21.15.83:0/281931422 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 803784945 0 0) 0x7f0354004eb0 con 0x7f036813f710 2024-03-25T19:20:37.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.371+0000 7f034ffff640 1 -- 172.21.15.83:0/281931422 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f03681397c0 con 0x7f036813f710 2024-03-25T19:20:37.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.371+0000 7f034ffff640 1 -- 172.21.15.83:0/281931422 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f035c001bc0 con 0x7f036813f710 2024-03-25T19:20:37.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.371+0000 7f034ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:37.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.371+0000 7f034ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:20:37.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.371+0000 7f034ffff640 10 monclient(hunting): dump: 2024-03-25T19:20:37.371 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:20:37.371 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:20:37.371 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:37.371 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:37.371 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:37.371 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:20:37.371 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:20:37.371 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:20:37.372 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:20:37.372 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:20:37.372 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.371+0000 7f034ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:20:37.380 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.380+0000 7f034ffff640 1 -- 172.21.15.83:0/281931422 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 170918608 0 0) 0x7f03681daa20 con 0x7f03680a98f0 2024-03-25T19:20:37.380 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.380+0000 7f034ffff640 1 -- 172.21.15.83:0/281931422 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0354004eb0 con 0x7f03680a98f0 2024-03-25T19:20:37.380 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.380+0000 7f034ffff640 1 -- 172.21.15.83:0/281931422 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f03580018b0 con 0x7f03680a98f0 2024-03-25T19:20:37.380 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.380+0000 7f034ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:37.380 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.380+0000 7f034ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:20:37.380 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.380+0000 7f034ffff640 10 monclient(hunting): dump: 2024-03-25T19:20:37.380 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:20:37.380 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:20:37.380 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:37.380 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:37.380 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:37.381 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:20:37.381 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:20:37.381 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:20:37.381 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:20:37.381 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:20:37.386 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.386+0000 7f034ffff640 1 -- 172.21.15.83:0/281931422 <== mon.1 v1:172.21.15.103:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3663707254 0 0) 0x7f03681397c0 con 0x7f036813f710 2024-03-25T19:20:37.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.386+0000 7f034ffff640 1 -- 172.21.15.83:0/281931422 >> v1:172.21.15.83:6790/0 conn(0x7f03680a98f0 legacy=0x7f03681daf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:37.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.386+0000 7f034ffff640 1 -- 172.21.15.83:0/281931422 >> v1:172.21.15.83:6789/0 conn(0x7f03681430c0 legacy=0x7f03681d7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:37.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.386+0000 7f034ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:20:37.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.386+0000 7f034ffff640 1 monclient: found mon.b 2024-03-25T19:20:37.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.386+0000 7f034ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:37.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.386+0000 7f034ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:20:37.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.386+0000 7f034ffff640 1 -- 172.21.15.83:0/281931422 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03681db650 con 0x7f036813f710 2024-03-25T19:20:37.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.386+0000 7f034ffff640 10 monclient: _finish_auth 0 2024-03-25T19:20:37.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.386+0000 7f034ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:20:37.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.386+0000 7f034ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:37.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.386+0000 7f036d896640 5 monclient: authenticate success, global_id 4264 2024-03-25T19:20:37.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.386+0000 7f036d896640 10 monclient: _renew_subs 2024-03-25T19:20:37.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.386+0000 7f036d896640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:20:37.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.386+0000 7f036d896640 1 -- 172.21.15.83:0/281931422 --> v1:172.21.15.103:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0368137720 con 0x7f036813f710 2024-03-25T19:20:37.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.387+0000 7f036d896640 10 client.4264.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:20:37.388 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.387+0000 7f036d896640 10 monclient: _renew_subs 2024-03-25T19:20:37.388 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.387+0000 7f036d896640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:20:37.388 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.387+0000 7f036d896640 1 -- 172.21.15.83:0/281931422 --> v1:172.21.15.103:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0368137d50 con 0x7f036813f710 2024-03-25T19:20:37.388 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.387+0000 7f036d896640 1 librados: init done 2024-03-25T19:20:37.388 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.387+0000 7f036d896640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:37.388 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.387+0000 7f036d896640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:37.389 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.387+0000 7f036d896640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:20:37.389 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.387+0000 7f036d896640 1 -- 172.21.15.83:0/281931422 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03681db840 con 0x7f036813f710 2024-03-25T19:20:37.679 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.679+0000 7f034ffff640 1 -- 172.21.15.83:0/281931422 <== mon.1 v1:172.21.15.103:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f035c0011f0 con 0x7f036813f710 2024-03-25T19:20:37.679 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.679+0000 7f034ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:37.679 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.679+0000 7f034ffff640 1 -- 172.21.15.83:0/281931422 <== mon.1 v1:172.21.15.103:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f035c003960 con 0x7f036813f710 2024-03-25T19:20:37.679 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.679+0000 7f034ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:37.680 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.679+0000 7f034ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:20:37.680 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.679+0000 7f034ffff640 10 monclient: dump: 2024-03-25T19:20:37.680 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:20:37.680 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:20:37.680 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:37.680 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:37.680 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:37.680 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:20:37.680 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:20:37.680 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:20:37.680 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:20:37.680 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:20:37.680 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.679+0000 7f03657fa640 4 set_mon_vals no callback set 2024-03-25T19:20:37.681 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.680+0000 7f034ffff640 1 -- 172.21.15.83:0/281931422 <== mon.1 v1:172.21.15.103:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7f035c003b40 con 0x7f036813f710 2024-03-25T19:20:37.681 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.681+0000 7f0366ffd640 10 monclient: get_auth_request con 0x7f035406df70 auth_method 0 2024-03-25T19:20:37.681 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.681+0000 7f034ffff640 10 client.4264.objecter ms_handle_connect 0x7f035406df70 2024-03-25T19:20:37.686 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.686+0000 7f034ffff640 1 -- 172.21.15.83:0/281931422 <== mon.1 v1:172.21.15.103:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7f035c054760 con 0x7f036813f710 2024-03-25T19:20:37.686 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.686+0000 7f034ffff640 10 client.4264.objecter ms_dispatch 0x7f0368000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:20:37.686 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.686+0000 7f034ffff640 3 client.4264.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:20:37.686 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.686+0000 7f034ffff640 3 client.4264.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:20:37.686 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.686+0000 7f034ffff640 20 client.4264.objecter dump_active .. 0 homeless 2024-03-25T19:20:37.686 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.687+0000 7f034ffff640 1 -- 172.21.15.83:0/281931422 <== mon.1 v1:172.21.15.103:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f035c050270 con 0x7f036813f710 2024-03-25T19:20:37.687 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.687+0000 7f034ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:37.687 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.687+0000 7f034ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:20:37.927 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.927+0000 7f036d896640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:37.927 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.927+0000 7f036d896640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:37.927 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.927+0000 7f036d896640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:20:37.927 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.927+0000 7f036d896640 1 -- 172.21.15.83:0/281931422 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0368074cc0 con 0x7f036813f710 2024-03-25T19:20:37.965 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.966+0000 7f034ffff640 1 -- 172.21.15.83:0/281931422 <== mon.1 v1:172.21.15.103:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7f035c0521a0 con 0x7f036813f710 2024-03-25T19:20:37.965 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.966+0000 7f034ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:37.965 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.966+0000 7f034ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:20:37.966 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:20:37.966 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:20:37.968 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.968+0000 7f036d896640 10 librados: watch_flush enter 2024-03-25T19:20:37.968 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.968+0000 7f036d896640 10 librados: watch_flush exit 2024-03-25T19:20:37.968 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.968+0000 7f036d896640 20 client.4264.objecter shutdown clearing up homeless session... 2024-03-25T19:20:37.968 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.968+0000 7f036d896640 10 client.4264.objecter successfully canceled tick 2024-03-25T19:20:37.968 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.968+0000 7f036d896640 1 -- 172.21.15.83:0/281931422 >> v1:172.21.15.103:6800/345072327 conn(0x7f035406df70 legacy=0x7f0354070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:37.969 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.968+0000 7f036d896640 10 monclient: shutdown 2024-03-25T19:20:37.969 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.968+0000 7f036d896640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:37.969 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.968+0000 7f036d896640 1 -- 172.21.15.83:0/281931422 >> v1:172.21.15.103:6789/0 conn(0x7f036813f710 legacy=0x7f03681d73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:37.969 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.968+0000 7f036d896640 1 -- 172.21.15.83:0/281931422 shutdown_connections 2024-03-25T19:20:37.969 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.968+0000 7f036d896640 1 -- 172.21.15.83:0/281931422 >> 172.21.15.83:0/281931422 conn(0x7f03680b7ea0 msgr2=0x7f036813c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:37.969 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.968+0000 7f036d896640 1 -- 172.21.15.83:0/281931422 shutdown_connections 2024-03-25T19:20:37.969 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.968+0000 7f036d896640 1 -- 172.21.15.83:0/281931422 wait complete. 2024-03-25T19:20:37.969 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:37.968+0000 7f036d896640 1 librados: shutdown 2024-03-25T19:20:37.980 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:20:42.268 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:01:24.195 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:20:42.335 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:01:24.262 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:20:42.343 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:01:24.253 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:20:42.387 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:01:24.315 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:20:42.404 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:01:24.314 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:20:42.463 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:01:24.391 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:20:42.582 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:01:24.509 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:20:42.626 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:01:24.536 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:20:42.710 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:01:24.637 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:20:44.786 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:20:44.901 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.901+0000 7ff50b24a640 10 monclient: get_monmap_and_config 2024-03-25T19:20:44.901 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.901+0000 7ff50b24a640 10 monclient: build_initial_monmap 2024-03-25T19:20:44.902 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.902+0000 7ff50b24a640 10 monclient: monmap: 2024-03-25T19:20:44.902 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:20:44.902 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:20:44.902 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:20:44.902904+0000 2024-03-25T19:20:44.902 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:20:44.902904+0000 2024-03-25T19:20:44.902 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:44.902 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:20:44.902 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:20:44.902 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:20:44.902 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:20:44.902 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:20:44.903 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.902+0000 7ff50b24a640 1 Processor -- start 2024-03-25T19:20:44.903 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.902+0000 7ff50b24a640 1 -- start start 2024-03-25T19:20:44.904 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.902+0000 7ff50b24a640 10 monclient: init 2024-03-25T19:20:44.904 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.903+0000 7ff50b24a640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:44.904 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.903+0000 7ff50b24a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:20:44.904 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.903+0000 7ff50b24a640 10 monclient(hunting): picked mon.noname-c con 0x7ff5040b4d60 addr v1:172.21.15.103:6789/0 2024-03-25T19:20:44.904 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.903+0000 7ff50b24a640 10 monclient(hunting): picked mon.noname-b con 0x7ff5040b1730 addr v1:172.21.15.83:6790/0 2024-03-25T19:20:44.904 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.903+0000 7ff50b24a640 10 monclient(hunting): picked mon.noname-a con 0x7ff50414be90 addr v1:172.21.15.83:6789/0 2024-03-25T19:20:44.904 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.903+0000 7ff50b24a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5040af940 con 0x7ff50414be90 2024-03-25T19:20:44.904 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.903+0000 7ff50b24a640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5040b0a70 con 0x7ff5040b4d60 2024-03-25T19:20:44.904 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.903+0000 7ff50b24a640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5040b0d50 con 0x7ff5040b1730 2024-03-25T19:20:44.904 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.903+0000 7ff50b24a640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:44.904 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.903+0000 7ff50b24a640 10 monclient(hunting): authenticate will time out at 6102.175505s 2024-03-25T19:20:44.912 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.913+0000 7ff508fe6640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7ff5040b4d60 0x7ff50413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:42262/0 (socket says 172.21.15.83:42262) 2024-03-25T19:20:44.912 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.913+0000 7ff508fe6640 1 -- 172.21.15.83:0/1934752983 learned_addr learned my addr 172.21.15.83:0/1934752983 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:20:44.928 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.929+0000 7ff4fb7fe640 1 -- 172.21.15.83:0/1934752983 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3955371113 0 0) 0x7ff5040b0a70 con 0x7ff5040b4d60 2024-03-25T19:20:44.928 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.929+0000 7ff4fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:20:44.928 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.929+0000 7ff4fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:44.929 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.929+0000 7ff4fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:44.929 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.929+0000 7ff4fb7fe640 10 monclient(hunting): my global_id is 4267 2024-03-25T19:20:44.929 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.929+0000 7ff4fb7fe640 1 -- 172.21.15.83:0/1934752983 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4ec003520 con 0x7ff5040b4d60 2024-03-25T19:20:44.932 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.933+0000 7ff4fb7fe640 1 -- 172.21.15.83:0/1934752983 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1502376785 0 0) 0x7ff5040b0d50 con 0x7ff5040b1730 2024-03-25T19:20:44.932 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.933+0000 7ff4fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:20:44.932 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.933+0000 7ff4fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:44.932 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.933+0000 7ff4fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:44.933 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.933+0000 7ff4fb7fe640 10 monclient(hunting): my global_id is 4271 2024-03-25T19:20:44.933 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.933+0000 7ff4fb7fe640 1 -- 172.21.15.83:0/1934752983 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5040b0a70 con 0x7ff5040b1730 2024-03-25T19:20:44.949 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.949+0000 7ff4fb7fe640 1 -- 172.21.15.83:0/1934752983 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3202204481 0 0) 0x7ff5040af940 con 0x7ff50414be90 2024-03-25T19:20:44.949 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.949+0000 7ff4fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:20:44.949 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.949+0000 7ff4fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:44.949 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.949+0000 7ff4fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:44.949 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.949+0000 7ff4fb7fe640 10 monclient(hunting): my global_id is 4260 2024-03-25T19:20:44.949 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.949+0000 7ff4fb7fe640 1 -- 172.21.15.83:0/1934752983 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5040b0d50 con 0x7ff50414be90 2024-03-25T19:20:44.952 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.952+0000 7ff4fb7fe640 1 -- 172.21.15.83:0/1934752983 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2396627606 0 0) 0x7ff4ec003520 con 0x7ff5040b4d60 2024-03-25T19:20:44.952 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.952+0000 7ff4fb7fe640 1 -- 172.21.15.83:0/1934752983 >> v1:172.21.15.83:6790/0 conn(0x7ff5040b1730 legacy=0x7ff5040b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:44.952 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.952+0000 7ff4fb7fe640 1 -- 172.21.15.83:0/1934752983 >> v1:172.21.15.83:6789/0 conn(0x7ff50414be90 legacy=0x7ff50414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:44.953 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.952+0000 7ff4fb7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:20:44.953 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.952+0000 7ff4fb7fe640 1 monclient: found mon.noname-c 2024-03-25T19:20:44.953 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.952+0000 7ff4fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:44.953 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.952+0000 7ff4fb7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.103:6789/0 2024-03-25T19:20:44.953 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.952+0000 7ff4fb7fe640 1 -- 172.21.15.83:0/1934752983 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5040b1030 con 0x7ff5040b4d60 2024-03-25T19:20:44.953 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.952+0000 7ff4fb7fe640 10 monclient: _finish_auth 0 2024-03-25T19:20:44.953 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.952+0000 7ff4fb7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:20:44.953 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.952+0000 7ff4fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:44.953 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.952+0000 7ff4fb7fe640 1 -- 172.21.15.83:0/1934752983 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7ff500001690 con 0x7ff5040b4d60 2024-03-25T19:20:44.953 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.952+0000 7ff4fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:44.953 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.952+0000 7ff4fb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:20:44.953 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.952+0000 7ff4fb7fe640 10 monclient: dump: 2024-03-25T19:20:44.953 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:20:44.953 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:20:44.954 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:44.954 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:44.954 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:44.954 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:20:44.954 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:20:44.954 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:20:44.954 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:20:44.954 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:20:44.954 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.952+0000 7ff50b24a640 5 monclient: authenticate success, global_id 4267 2024-03-25T19:20:44.954 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.952+0000 7ff50b24a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:20:44.959 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.959+0000 7ff4fb7fe640 1 -- 172.21.15.83:0/1934752983 <== mon.1 v1:172.21.15.103:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff5000017f0 con 0x7ff5040b4d60 2024-03-25T19:20:44.959 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.960+0000 7ff4fb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:44.959 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.960+0000 7ff4fb7fe640 1 -- 172.21.15.83:0/1934752983 <== mon.1 v1:172.21.15.103:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7ff500001c00 con 0x7ff5040b4d60 2024-03-25T19:20:44.960 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.960+0000 7ff50b24a640 10 monclient: get_monmap_and_config success 2024-03-25T19:20:44.960 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.960+0000 7ff50b24a640 4 set_mon_vals no callback set 2024-03-25T19:20:44.960 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.960+0000 7ff4fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:44.960 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.960+0000 7ff4fb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:20:44.960 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.960+0000 7ff4fb7fe640 10 monclient: dump: 2024-03-25T19:20:44.960 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:20:44.960 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:20:44.960 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:44.960 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:44.960 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:44.960 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:20:44.960 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:20:44.960 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:20:44.961 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:20:44.961 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:20:44.963 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.960+0000 7ff50b24a640 10 monclient: shutdown 2024-03-25T19:20:44.963 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.960+0000 7ff50b24a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:44.963 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.960+0000 7ff50b24a640 1 -- 172.21.15.83:0/1934752983 >> v1:172.21.15.103:6789/0 conn(0x7ff5040b4d60 legacy=0x7ff50413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:44.963 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.960+0000 7ff50b24a640 1 -- 172.21.15.83:0/1934752983 shutdown_connections 2024-03-25T19:20:44.963 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.960+0000 7ff50b24a640 1 -- 172.21.15.83:0/1934752983 >> 172.21.15.83:0/1934752983 conn(0x7ff5040617a0 msgr2=0x7ff504061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:44.963 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.960+0000 7ff50b24a640 1 -- 172.21.15.83:0/1934752983 shutdown_connections 2024-03-25T19:20:44.963 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.960+0000 7ff50b24a640 1 -- 172.21.15.83:0/1934752983 wait complete. 2024-03-25T19:20:44.963 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.961+0000 7ff50b24a640 10 monclient: build_initial_monmap 2024-03-25T19:20:44.963 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.961+0000 7ff50b24a640 10 monclient: monmap: 2024-03-25T19:20:44.964 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:20:44.964 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:20:44.964 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:20:44.964 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:20:44.964 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:44.964 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:20:44.964 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:20:44.964 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:20:44.964 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:20:44.964 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:20:44.964 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.961+0000 7ff50b24a640 1 librados: starting msgr at 2024-03-25T19:20:44.964 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.961+0000 7ff50b24a640 1 librados: starting objecter 2024-03-25T19:20:44.964 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.962+0000 7ff50b24a640 1 Processor -- start 2024-03-25T19:20:44.964 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.962+0000 7ff50b24a640 1 -- start start 2024-03-25T19:20:44.964 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.962+0000 7ff50b24a640 1 librados: setting wanted keys 2024-03-25T19:20:44.964 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.962+0000 7ff50b24a640 1 librados: calling monclient init 2024-03-25T19:20:44.965 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.962+0000 7ff50b24a640 10 monclient: init 2024-03-25T19:20:44.965 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.962+0000 7ff50b24a640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:44.965 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.962+0000 7ff50b24a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:20:44.965 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.962+0000 7ff50b24a640 10 monclient(hunting): picked mon.noname-b con 0x7ff5040b1730 addr v1:172.21.15.103:6789/0 2024-03-25T19:20:44.965 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.962+0000 7ff50b24a640 10 monclient(hunting): picked mon.noname-a con 0x7ff5040b4d60 addr v1:172.21.15.83:6789/0 2024-03-25T19:20:44.965 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.962+0000 7ff50b24a640 10 monclient(hunting): picked mon.noname-c con 0x7ff50414be90 addr v1:172.21.15.83:6790/0 2024-03-25T19:20:44.965 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.962+0000 7ff50b24a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff504140ce0 con 0x7ff5040b4d60 2024-03-25T19:20:44.965 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.962+0000 7ff50b24a640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5041d7a00 con 0x7ff5040b1730 2024-03-25T19:20:44.965 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.962+0000 7ff50b24a640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5041d85a0 con 0x7ff50414be90 2024-03-25T19:20:44.965 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.962+0000 7ff50b24a640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:44.965 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.962+0000 7ff50b24a640 10 monclient(hunting): authenticate will time out at 6102.234552s 2024-03-25T19:20:44.965 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.963+0000 7ff5097e7640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7ff50414be90 0x7ff504142cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:39284/0 (socket says 172.21.15.83:39284) 2024-03-25T19:20:44.965 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.963+0000 7ff5097e7640 1 -- 172.21.15.83:0/3765476890 learned_addr learned my addr 172.21.15.83:0/3765476890 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:20:44.965 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.963+0000 7ff4f97fa640 10 client.?.objecter ms_handle_connect 0x7ff50414be90 2024-03-25T19:20:44.965 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.963+0000 7ff4f97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:44.966 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.965+0000 7ff4f97fa640 10 client.?.objecter ms_handle_connect 0x7ff5040b1730 2024-03-25T19:20:44.966 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:44.965+0000 7ff4f97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:45.029 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.030+0000 7ff4f97fa640 10 client.?.objecter ms_handle_connect 0x7ff5040b4d60 2024-03-25T19:20:45.030 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.030+0000 7ff4f97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:45.036 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.036+0000 7ff4f97fa640 1 -- 172.21.15.83:0/3765476890 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3910735494 0 0) 0x7ff5040b0a70 con 0x7ff5040b4d60 2024-03-25T19:20:45.036 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.036+0000 7ff4f97fa640 10 monclient(hunting): _negotiate 2024-03-25T19:20:45.036 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.036+0000 7ff4f97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:45.036 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.036+0000 7ff4f97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:45.036 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.036+0000 7ff4f97fa640 10 monclient(hunting): my global_id is 4263 2024-03-25T19:20:45.036 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.036+0000 7ff4f97fa640 1 -- 172.21.15.83:0/3765476890 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4f0004eb0 con 0x7ff5040b4d60 2024-03-25T19:20:45.052 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.053+0000 7ff4f97fa640 1 -- 172.21.15.83:0/3765476890 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 983669299 0 0) 0x7ff4f0004eb0 con 0x7ff5040b4d60 2024-03-25T19:20:45.053 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.053+0000 7ff4f97fa640 1 -- 172.21.15.83:0/3765476890 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff5040b0a70 con 0x7ff5040b4d60 2024-03-25T19:20:45.053 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.053+0000 7ff4f97fa640 1 -- 172.21.15.83:0/3765476890 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7ff4f40017d0 con 0x7ff5040b4d60 2024-03-25T19:20:45.053 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.053+0000 7ff4f97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:45.053 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.053+0000 7ff4f97fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:20:45.053 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.053+0000 7ff4f97fa640 10 monclient(hunting): dump: 2024-03-25T19:20:45.053 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:20:45.053 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:20:45.053 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:45.053 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:45.053 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:45.053 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:20:45.053 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:20:45.053 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:20:45.053 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:20:45.053 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:20:45.054 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.053+0000 7ff4f97fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:20:45.069 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.069+0000 7ff4f97fa640 1 -- 172.21.15.83:0/3765476890 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1013413584 0 0) 0x7ff5040b0a70 con 0x7ff5040b4d60 2024-03-25T19:20:45.070 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.069+0000 7ff4f97fa640 1 -- 172.21.15.83:0/3765476890 >> v1:172.21.15.83:6790/0 conn(0x7ff50414be90 legacy=0x7ff504142cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:45.070 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.069+0000 7ff4f97fa640 1 -- 172.21.15.83:0/3765476890 >> v1:172.21.15.103:6789/0 conn(0x7ff5040b1730 legacy=0x7ff5041d7e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:45.070 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.069+0000 7ff4f97fa640 10 monclient: _finish_hunting 0 2024-03-25T19:20:45.070 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.069+0000 7ff4f97fa640 1 monclient: found mon.a 2024-03-25T19:20:45.070 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.069+0000 7ff4f97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:45.070 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.069+0000 7ff4f97fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:20:45.070 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.069+0000 7ff4f97fa640 1 -- 172.21.15.83:0/3765476890 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5041d7690 con 0x7ff5040b4d60 2024-03-25T19:20:45.070 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.070+0000 7ff4f97fa640 10 monclient: _finish_auth 0 2024-03-25T19:20:45.070 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.070+0000 7ff4f97fa640 10 monclient: _check_auth_tickets 2024-03-25T19:20:45.070 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.070+0000 7ff4f97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:45.070 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.070+0000 7ff50b24a640 5 monclient: authenticate success, global_id 4263 2024-03-25T19:20:45.070 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.070+0000 7ff50b24a640 10 monclient: _renew_subs 2024-03-25T19:20:45.070 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.070+0000 7ff50b24a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:20:45.070 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.070+0000 7ff50b24a640 1 -- 172.21.15.83:0/3765476890 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff50413bcc0 con 0x7ff5040b4d60 2024-03-25T19:20:45.071 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.070+0000 7ff50b24a640 10 client.4263.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:20:45.071 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.070+0000 7ff50b24a640 10 monclient: _renew_subs 2024-03-25T19:20:45.071 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.070+0000 7ff50b24a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:20:45.071 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.070+0000 7ff50b24a640 1 -- 172.21.15.83:0/3765476890 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5041405f0 con 0x7ff5040b4d60 2024-03-25T19:20:45.071 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.070+0000 7ff50b24a640 1 librados: init done 2024-03-25T19:20:45.071 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.071+0000 7ff50b24a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:45.071 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.071+0000 7ff50b24a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:45.072 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.071+0000 7ff50b24a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:20:45.072 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.071+0000 7ff50b24a640 1 -- 172.21.15.83:0/3765476890 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5040627b0 con 0x7ff5040b4d60 2024-03-25T19:20:45.426 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.426+0000 7ff4f97fa640 1 -- 172.21.15.83:0/3765476890 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4f4001d20 con 0x7ff5040b4d60 2024-03-25T19:20:45.426 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.426+0000 7ff4f97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:45.426 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.426+0000 7ff4f97fa640 1 -- 172.21.15.83:0/3765476890 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7ff4f40037b0 con 0x7ff5040b4d60 2024-03-25T19:20:45.426 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.426+0000 7ff4f97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:45.426 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.426+0000 7ff4f97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:20:45.427 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.426+0000 7ff4f97fa640 10 monclient: dump: 2024-03-25T19:20:45.427 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:20:45.427 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:20:45.427 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:45.427 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:45.427 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:45.427 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:20:45.427 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:20:45.427 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:20:45.427 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:20:45.427 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:20:45.427 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.426+0000 7ff4faffd640 4 set_mon_vals no callback set 2024-03-25T19:20:45.428 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.427+0000 7ff4f97fa640 1 -- 172.21.15.83:0/3765476890 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7ff4f4019800 con 0x7ff5040b4d60 2024-03-25T19:20:45.428 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.428+0000 7ff508fe6640 10 monclient: get_auth_request con 0x7ff4f006dc40 auth_method 0 2024-03-25T19:20:45.428 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.428+0000 7ff4f97fa640 10 client.4263.objecter ms_handle_connect 0x7ff4f006dc40 2024-03-25T19:20:45.432 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.432+0000 7ff4f97fa640 1 -- 172.21.15.83:0/3765476890 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7ff4f4053560 con 0x7ff5040b4d60 2024-03-25T19:20:45.432 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.432+0000 7ff4f97fa640 10 client.4263.objecter ms_dispatch 0x7ff504000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:20:45.432 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.432+0000 7ff4f97fa640 3 client.4263.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:20:45.432 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.432+0000 7ff4f97fa640 3 client.4263.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:20:45.432 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.432+0000 7ff4f97fa640 20 client.4263.objecter dump_active .. 0 homeless 2024-03-25T19:20:45.432 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.432+0000 7ff4f97fa640 1 -- 172.21.15.83:0/3765476890 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff4f404eea0 con 0x7ff5040b4d60 2024-03-25T19:20:45.433 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.432+0000 7ff4f97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:45.433 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.432+0000 7ff4f97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:20:45.692 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.692+0000 7ff50b24a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:45.692 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.692+0000 7ff50b24a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:45.692 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.692+0000 7ff50b24a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:20:45.693 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.692+0000 7ff50b24a640 1 -- 172.21.15.83:0/3765476890 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff5041d7820 con 0x7ff5040b4d60 2024-03-25T19:20:45.740 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.741+0000 7ff4f97fa640 1 -- 172.21.15.83:0/3765476890 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7ff4f4050fa0 con 0x7ff5040b4d60 2024-03-25T19:20:45.740 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.741+0000 7ff4f97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:45.740 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.741+0000 7ff4f97fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:20:45.741 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:20:45.741 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:20:45.743 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.743+0000 7ff50b24a640 10 librados: watch_flush enter 2024-03-25T19:20:45.743 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.743+0000 7ff50b24a640 10 librados: watch_flush exit 2024-03-25T19:20:45.743 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.743+0000 7ff50b24a640 20 client.4263.objecter shutdown clearing up homeless session... 2024-03-25T19:20:45.744 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.743+0000 7ff50b24a640 10 client.4263.objecter successfully canceled tick 2024-03-25T19:20:45.744 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.743+0000 7ff50b24a640 1 -- 172.21.15.83:0/3765476890 >> v1:172.21.15.103:6800/345072327 conn(0x7ff4f006dc40 legacy=0x7ff4f0070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:45.744 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.743+0000 7ff50b24a640 10 monclient: shutdown 2024-03-25T19:20:45.744 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.743+0000 7ff50b24a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:45.744 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.743+0000 7ff50b24a640 1 -- 172.21.15.83:0/3765476890 >> v1:172.21.15.83:6789/0 conn(0x7ff5040b4d60 legacy=0x7ff50413b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:45.744 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.743+0000 7ff50b24a640 1 -- 172.21.15.83:0/3765476890 shutdown_connections 2024-03-25T19:20:45.744 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.743+0000 7ff50b24a640 1 -- 172.21.15.83:0/3765476890 >> 172.21.15.83:0/3765476890 conn(0x7ff5040617a0 msgr2=0x7ff504061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:45.744 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.743+0000 7ff50b24a640 1 -- 172.21.15.83:0/3765476890 shutdown_connections 2024-03-25T19:20:45.744 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.743+0000 7ff50b24a640 1 -- 172.21.15.83:0/3765476890 wait complete. 2024-03-25T19:20:45.744 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:45.743+0000 7ff50b24a640 1 librados: shutdown 2024-03-25T19:20:45.755 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:20:52.561 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:20:52.677 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.676+0000 7f3e4020f640 10 monclient: get_monmap_and_config 2024-03-25T19:20:52.677 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.676+0000 7f3e4020f640 10 monclient: build_initial_monmap 2024-03-25T19:20:52.677 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.676+0000 7f3e4020f640 10 monclient: monmap: 2024-03-25T19:20:52.677 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:20:52.677 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:20:52.677 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:20:52.678384+0000 2024-03-25T19:20:52.677 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:20:52.678384+0000 2024-03-25T19:20:52.677 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:52.677 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:20:52.677 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:20:52.677 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:20:52.678 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:20:52.678 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:20:52.679 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.677+0000 7f3e4020f640 1 Processor -- start 2024-03-25T19:20:52.679 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.677+0000 7f3e4020f640 1 -- start start 2024-03-25T19:20:52.679 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.677+0000 7f3e4020f640 10 monclient: init 2024-03-25T19:20:52.679 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.677+0000 7f3e4020f640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:52.679 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.677+0000 7f3e4020f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:20:52.679 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.677+0000 7f3e4020f640 10 monclient(hunting): picked mon.noname-b con 0x7f3e3806c380 addr v1:172.21.15.83:6790/0 2024-03-25T19:20:52.679 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.678+0000 7f3e4020f640 10 monclient(hunting): picked mon.noname-c con 0x7f3e380b12d0 addr v1:172.21.15.103:6789/0 2024-03-25T19:20:52.679 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.678+0000 7f3e4020f640 10 monclient(hunting): picked mon.noname-a con 0x7f3e380aedf0 addr v1:172.21.15.83:6789/0 2024-03-25T19:20:52.679 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.678+0000 7f3e4020f640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e380b16f0 con 0x7f3e380aedf0 2024-03-25T19:20:52.679 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.678+0000 7f3e4020f640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e380adfe0 con 0x7f3e380b12d0 2024-03-25T19:20:52.679 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.678+0000 7f3e4020f640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e380add00 con 0x7f3e3806c380 2024-03-25T19:20:52.680 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.678+0000 7f3e4020f640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:52.680 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.678+0000 7f3e4020f640 10 monclient(hunting): authenticate will time out at 6109.950820s 2024-03-25T19:20:52.694 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.693+0000 7f3e3d7aa640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f3e380b12d0 0x7f3e380ae6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:58582/0 (socket says 172.21.15.83:58582) 2024-03-25T19:20:52.694 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.693+0000 7f3e3d7aa640 1 -- 172.21.15.83:0/3015075204 learned_addr learned my addr 172.21.15.83:0/3015075204 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:20:52.705 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.704+0000 7f3e3cfa9640 1 -- 172.21.15.83:0/3015075204 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1206954089 0 0) 0x7f3e380add00 con 0x7f3e3806c380 2024-03-25T19:20:52.705 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.704+0000 7f3e3cfa9640 10 monclient(hunting): _negotiate 2024-03-25T19:20:52.705 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.704+0000 7f3e3cfa9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:52.705 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.704+0000 7f3e3cfa9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:52.705 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.704+0000 7f3e3cfa9640 10 monclient(hunting): my global_id is 4277 2024-03-25T19:20:52.705 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.704+0000 7f3e3cfa9640 1 -- 172.21.15.83:0/3015075204 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e24003520 con 0x7f3e3806c380 2024-03-25T19:20:52.712 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.711+0000 7f3e3cfa9640 1 -- 172.21.15.83:0/3015075204 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 286410097 0 0) 0x7f3e380b16f0 con 0x7f3e380aedf0 2024-03-25T19:20:52.712 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.711+0000 7f3e3cfa9640 10 monclient(hunting): _negotiate 2024-03-25T19:20:52.712 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.711+0000 7f3e3cfa9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:52.712 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.711+0000 7f3e3cfa9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:52.712 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.711+0000 7f3e3cfa9640 10 monclient(hunting): my global_id is 4266 2024-03-25T19:20:52.712 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.711+0000 7f3e3cfa9640 1 -- 172.21.15.83:0/3015075204 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e380add00 con 0x7f3e380aedf0 2024-03-25T19:20:52.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.715+0000 7f3e3cfa9640 1 -- 172.21.15.83:0/3015075204 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2126157894 0 0) 0x7f3e380adfe0 con 0x7f3e380b12d0 2024-03-25T19:20:52.716 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.715+0000 7f3e3cfa9640 10 monclient(hunting): _negotiate 2024-03-25T19:20:52.716 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.715+0000 7f3e3cfa9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:52.716 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.715+0000 7f3e3cfa9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:52.716 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.715+0000 7f3e3cfa9640 10 monclient(hunting): my global_id is 4273 2024-03-25T19:20:52.716 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.715+0000 7f3e3cfa9640 1 -- 172.21.15.83:0/3015075204 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e380b16f0 con 0x7f3e380b12d0 2024-03-25T19:20:52.725 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.724+0000 7f3e3cfa9640 1 -- 172.21.15.83:0/3015075204 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2442114923 0 0) 0x7f3e380add00 con 0x7f3e380aedf0 2024-03-25T19:20:52.725 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.724+0000 7f3e3cfa9640 1 -- 172.21.15.83:0/3015075204 >> v1:172.21.15.83:6790/0 conn(0x7f3e3806c380 legacy=0x7f3e380b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:52.725 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.724+0000 7f3e3cfa9640 1 -- 172.21.15.83:0/3015075204 >> v1:172.21.15.103:6789/0 conn(0x7f3e380b12d0 legacy=0x7f3e380ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:52.725 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.724+0000 7f3e3cfa9640 10 monclient: _finish_hunting 0 2024-03-25T19:20:52.725 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.724+0000 7f3e3cfa9640 1 monclient: found mon.noname-a 2024-03-25T19:20:52.725 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.724+0000 7f3e3cfa9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:52.725 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.724+0000 7f3e3cfa9640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.83:6789/0 2024-03-25T19:20:52.725 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.724+0000 7f3e3cfa9640 1 -- 172.21.15.83:0/3015075204 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e380ae2c0 con 0x7f3e380aedf0 2024-03-25T19:20:52.725 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.724+0000 7f3e3cfa9640 10 monclient: _finish_auth 0 2024-03-25T19:20:52.725 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.724+0000 7f3e3cfa9640 10 monclient: _check_auth_tickets 2024-03-25T19:20:52.725 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.724+0000 7f3e3cfa9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:52.725 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.724+0000 7f3e3cfa9640 1 -- 172.21.15.83:0/3015075204 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f3e300019f0 con 0x7f3e380aedf0 2024-03-25T19:20:52.726 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.724+0000 7f3e3cfa9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:52.726 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.724+0000 7f3e3cfa9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:20:52.726 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.724+0000 7f3e3cfa9640 10 monclient: dump: 2024-03-25T19:20:52.726 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:20:52.726 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:20:52.726 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:52.726 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:52.726 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:52.726 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:20:52.726 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:20:52.726 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:20:52.726 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:20:52.726 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:20:52.726 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.724+0000 7f3e4020f640 5 monclient: authenticate success, global_id 4266 2024-03-25T19:20:52.726 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.724+0000 7f3e4020f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:20:52.748 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.748+0000 7f3e3cfa9640 1 -- 172.21.15.83:0/3015075204 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e30001b50 con 0x7f3e380aedf0 2024-03-25T19:20:52.748 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.748+0000 7f3e3cfa9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:52.749 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.748+0000 7f3e3cfa9640 1 -- 172.21.15.83:0/3015075204 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f3e30001ee0 con 0x7f3e380aedf0 2024-03-25T19:20:52.749 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.748+0000 7f3e3cfa9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:52.749 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.748+0000 7f3e3cfa9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:20:52.749 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.748+0000 7f3e3cfa9640 10 monclient: dump: 2024-03-25T19:20:52.749 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:20:52.749 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:20:52.749 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:52.749 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:52.749 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:52.749 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:20:52.749 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:20:52.749 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:20:52.749 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:20:52.749 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:20:52.749 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.748+0000 7f3e4020f640 10 monclient: get_monmap_and_config success 2024-03-25T19:20:52.750 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.748+0000 7f3e4020f640 4 set_mon_vals no callback set 2024-03-25T19:20:52.751 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.749+0000 7f3e4020f640 10 monclient: shutdown 2024-03-25T19:20:52.751 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.749+0000 7f3e4020f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:52.751 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.749+0000 7f3e4020f640 1 -- 172.21.15.83:0/3015075204 >> v1:172.21.15.83:6789/0 conn(0x7f3e380aedf0 legacy=0x7f3e3814e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:52.751 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.749+0000 7f3e4020f640 1 -- 172.21.15.83:0/3015075204 shutdown_connections 2024-03-25T19:20:52.751 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.749+0000 7f3e4020f640 1 -- 172.21.15.83:0/3015075204 >> 172.21.15.83:0/3015075204 conn(0x7f3e380b5d10 msgr2=0x7f3e380b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:52.751 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.749+0000 7f3e4020f640 1 -- 172.21.15.83:0/3015075204 shutdown_connections 2024-03-25T19:20:52.752 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.749+0000 7f3e4020f640 1 -- 172.21.15.83:0/3015075204 wait complete. 2024-03-25T19:20:52.752 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.750+0000 7f3e4020f640 10 monclient: build_initial_monmap 2024-03-25T19:20:52.752 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.750+0000 7f3e4020f640 10 monclient: monmap: 2024-03-25T19:20:52.752 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:20:52.752 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:20:52.752 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:20:52.752 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:20:52.752 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:52.752 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:20:52.752 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:20:52.752 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:20:52.752 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:20:52.752 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:20:52.752 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.750+0000 7f3e4020f640 1 librados: starting msgr at 2024-03-25T19:20:52.752 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.750+0000 7f3e4020f640 1 librados: starting objecter 2024-03-25T19:20:52.752 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.750+0000 7f3e4020f640 1 Processor -- start 2024-03-25T19:20:52.752 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.751+0000 7f3e4020f640 1 -- start start 2024-03-25T19:20:52.753 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.751+0000 7f3e4020f640 1 librados: setting wanted keys 2024-03-25T19:20:52.753 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.751+0000 7f3e4020f640 1 librados: calling monclient init 2024-03-25T19:20:52.753 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.751+0000 7f3e4020f640 10 monclient: init 2024-03-25T19:20:52.754 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.751+0000 7f3e4020f640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:52.754 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.751+0000 7f3e4020f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:20:52.754 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.751+0000 7f3e4020f640 10 monclient(hunting): picked mon.noname-b con 0x7f3e3806c380 addr v1:172.21.15.103:6789/0 2024-03-25T19:20:52.754 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.752+0000 7f3e4020f640 10 monclient(hunting): picked mon.noname-c con 0x7f3e380aedf0 addr v1:172.21.15.83:6790/0 2024-03-25T19:20:52.754 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.752+0000 7f3e4020f640 10 monclient(hunting): picked mon.noname-a con 0x7f3e380b12d0 addr v1:172.21.15.83:6789/0 2024-03-25T19:20:52.754 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.752+0000 7f3e4020f640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e3806c8e0 con 0x7f3e380b12d0 2024-03-25T19:20:52.754 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.752+0000 7f3e4020f640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e381d8150 con 0x7f3e3806c380 2024-03-25T19:20:52.754 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.752+0000 7f3e4020f640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e381d7e70 con 0x7f3e380aedf0 2024-03-25T19:20:52.754 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.752+0000 7f3e4020f640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:52.754 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.752+0000 7f3e4020f640 10 monclient(hunting): authenticate will time out at 6110.025030s 2024-03-25T19:20:52.763 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.762+0000 7f3e3e7ac640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f3e380b12d0 0x7f3e381c22e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49062/0 (socket says 172.21.15.83:49062) 2024-03-25T19:20:52.763 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.762+0000 7f3e3e7ac640 1 -- 172.21.15.83:0/965218504 learned_addr learned my addr 172.21.15.83:0/965218504 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:20:52.763 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.763+0000 7f3e2e7fc640 10 client.?.objecter ms_handle_connect 0x7f3e380b12d0 2024-03-25T19:20:52.763 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.763+0000 7f3e2e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:52.767 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.767+0000 7f3e2e7fc640 10 client.?.objecter ms_handle_connect 0x7f3e3806c380 2024-03-25T19:20:52.768 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.767+0000 7f3e2e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:52.768 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.767+0000 7f3e2e7fc640 10 client.?.objecter ms_handle_connect 0x7f3e380aedf0 2024-03-25T19:20:52.768 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.768+0000 7f3e2e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:52.779 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.779+0000 7f3e2e7fc640 1 -- 172.21.15.83:0/965218504 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3900365876 0 0) 0x7f3e381d8150 con 0x7f3e3806c380 2024-03-25T19:20:52.779 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.779+0000 7f3e2e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:20:52.779 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.779+0000 7f3e2e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:52.779 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.779+0000 7f3e2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:52.780 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.779+0000 7f3e2e7fc640 10 monclient(hunting): my global_id is 4276 2024-03-25T19:20:52.780 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.779+0000 7f3e2e7fc640 1 -- 172.21.15.83:0/965218504 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e1c004eb0 con 0x7f3e3806c380 2024-03-25T19:20:52.782 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.781+0000 7f3e2e7fc640 1 -- 172.21.15.83:0/965218504 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3527283037 0 0) 0x7f3e3806c8e0 con 0x7f3e380b12d0 2024-03-25T19:20:52.782 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.781+0000 7f3e2e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:20:52.782 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.781+0000 7f3e2e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:52.782 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.781+0000 7f3e2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:52.783 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.781+0000 7f3e2e7fc640 10 monclient(hunting): my global_id is 4269 2024-03-25T19:20:52.783 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.782+0000 7f3e2e7fc640 1 -- 172.21.15.83:0/965218504 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e381d8150 con 0x7f3e380b12d0 2024-03-25T19:20:52.785 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.784+0000 7f3e2e7fc640 1 -- 172.21.15.83:0/965218504 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 615551259 0 0) 0x7f3e381d7e70 con 0x7f3e380aedf0 2024-03-25T19:20:52.785 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.784+0000 7f3e2e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:20:52.785 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.784+0000 7f3e2e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:52.785 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.784+0000 7f3e2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:52.785 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.784+0000 7f3e2e7fc640 10 monclient(hunting): my global_id is 4280 2024-03-25T19:20:52.785 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.784+0000 7f3e2e7fc640 1 -- 172.21.15.83:0/965218504 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e3806c8e0 con 0x7f3e380aedf0 2024-03-25T19:20:52.789 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.788+0000 7f3e2e7fc640 1 -- 172.21.15.83:0/965218504 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 887420031 0 0) 0x7f3e1c004eb0 con 0x7f3e3806c380 2024-03-25T19:20:52.789 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.788+0000 7f3e2e7fc640 1 -- 172.21.15.83:0/965218504 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e381d7e70 con 0x7f3e3806c380 2024-03-25T19:20:52.789 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.788+0000 7f3e2e7fc640 1 -- 172.21.15.83:0/965218504 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f3e34001950 con 0x7f3e3806c380 2024-03-25T19:20:52.789 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.788+0000 7f3e2e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:52.789 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.788+0000 7f3e2e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:20:52.790 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.788+0000 7f3e2e7fc640 10 monclient(hunting): dump: 2024-03-25T19:20:52.790 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:20:52.790 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:20:52.790 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:52.790 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:52.790 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:52.790 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:20:52.790 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:20:52.790 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:20:52.790 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:20:52.790 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:20:52.790 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.788+0000 7f3e2e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:20:52.795 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.794+0000 7f3e2e7fc640 1 -- 172.21.15.83:0/965218504 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 65836751 0 0) 0x7f3e3806c8e0 con 0x7f3e380aedf0 2024-03-25T19:20:52.795 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.794+0000 7f3e2e7fc640 1 -- 172.21.15.83:0/965218504 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e1c004eb0 con 0x7f3e380aedf0 2024-03-25T19:20:52.795 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.794+0000 7f3e2e7fc640 1 -- 172.21.15.83:0/965218504 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f3e28001790 con 0x7f3e380aedf0 2024-03-25T19:20:52.795 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.794+0000 7f3e2e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:52.795 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.794+0000 7f3e2e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:20:52.795 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.794+0000 7f3e2e7fc640 10 monclient(hunting): dump: 2024-03-25T19:20:52.795 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:20:52.795 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:20:52.795 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:52.795 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:52.795 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:52.795 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:20:52.796 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:20:52.796 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:20:52.796 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:20:52.796 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:20:52.801 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.799+0000 7f3e2e7fc640 1 -- 172.21.15.83:0/965218504 <== mon.1 v1:172.21.15.103:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3329375472 0 0) 0x7f3e381d7e70 con 0x7f3e3806c380 2024-03-25T19:20:52.801 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.800+0000 7f3e2e7fc640 1 -- 172.21.15.83:0/965218504 >> v1:172.21.15.83:6790/0 conn(0x7f3e380aedf0 legacy=0x7f3e381c07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:52.801 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.800+0000 7f3e2e7fc640 1 -- 172.21.15.83:0/965218504 >> v1:172.21.15.83:6789/0 conn(0x7f3e380b12d0 legacy=0x7f3e381c22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:52.801 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.800+0000 7f3e2e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:20:52.801 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.800+0000 7f3e2e7fc640 1 monclient: found mon.b 2024-03-25T19:20:52.801 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.800+0000 7f3e2e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:52.801 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.800+0000 7f3e2e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:20:52.801 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.800+0000 7f3e2e7fc640 1 -- 172.21.15.83:0/965218504 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e381c19a0 con 0x7f3e3806c380 2024-03-25T19:20:52.801 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.800+0000 7f3e2e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:20:52.801 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.800+0000 7f3e2e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:20:52.801 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.800+0000 7f3e2e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:52.801 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.800+0000 7f3e4020f640 5 monclient: authenticate success, global_id 4276 2024-03-25T19:20:52.802 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.800+0000 7f3e4020f640 10 monclient: _renew_subs 2024-03-25T19:20:52.802 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.800+0000 7f3e4020f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:20:52.802 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.800+0000 7f3e4020f640 1 -- 172.21.15.83:0/965218504 --> v1:172.21.15.103:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e381c4d40 con 0x7f3e3806c380 2024-03-25T19:20:52.802 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.800+0000 7f3e4020f640 10 client.4276.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:20:52.802 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.800+0000 7f3e4020f640 10 monclient: _renew_subs 2024-03-25T19:20:52.802 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.800+0000 7f3e4020f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:20:52.802 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.800+0000 7f3e4020f640 1 -- 172.21.15.83:0/965218504 --> v1:172.21.15.103:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e381c10d0 con 0x7f3e3806c380 2024-03-25T19:20:52.802 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.800+0000 7f3e4020f640 1 librados: init done 2024-03-25T19:20:52.802 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.801+0000 7f3e4020f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:52.803 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.801+0000 7f3e4020f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:52.803 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.801+0000 7f3e4020f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:20:52.803 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.801+0000 7f3e4020f640 1 -- 172.21.15.83:0/965218504 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e381d82e0 con 0x7f3e3806c380 2024-03-25T19:20:52.805 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.804+0000 7f3e2e7fc640 1 -- 172.21.15.83:0/965218504 <== mon.1 v1:172.21.15.103:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e34003610 con 0x7f3e3806c380 2024-03-25T19:20:52.805 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.804+0000 7f3e2e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:52.805 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:52.804+0000 7f3e2ffff640 4 set_mon_vals no callback set 2024-03-25T19:20:53.080 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:53.080+0000 7f3e2e7fc640 1 -- 172.21.15.83:0/965218504 <== mon.1 v1:172.21.15.103:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f3e34001d80 con 0x7f3e3806c380 2024-03-25T19:20:53.081 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:53.080+0000 7f3e2e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:53.081 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:53.080+0000 7f3e2e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:20:53.081 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:53.080+0000 7f3e2e7fc640 10 monclient: dump: 2024-03-25T19:20:53.081 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:20:53.081 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:20:53.081 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:53.081 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:20:53.081 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:53.081 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:20:53.082 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:20:53.082 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:20:53.082 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:20:53.082 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:20:53.083 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:53.080+0000 7f3e2e7fc640 1 -- 172.21.15.83:0/965218504 <== mon.1 v1:172.21.15.103:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7f3e34003aa0 con 0x7f3e3806c380 2024-03-25T19:20:53.083 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:53.081+0000 7f3e3d7aa640 10 monclient: get_auth_request con 0x7f3e1c06df70 auth_method 0 2024-03-25T19:20:53.083 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:53.081+0000 7f3e2e7fc640 10 client.4276.objecter ms_handle_connect 0x7f3e1c06df70 2024-03-25T19:20:53.086 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:53.085+0000 7f3e2e7fc640 1 -- 172.21.15.83:0/965218504 <== mon.1 v1:172.21.15.103:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7f3e34004460 con 0x7f3e3806c380 2024-03-25T19:20:53.086 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:53.085+0000 7f3e2e7fc640 10 client.4276.objecter ms_dispatch 0x7f3e38000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:20:53.086 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:53.085+0000 7f3e2e7fc640 3 client.4276.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:20:53.086 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:53.085+0000 7f3e2e7fc640 3 client.4276.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:20:53.086 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:53.085+0000 7f3e2e7fc640 20 client.4276.objecter dump_active .. 0 homeless 2024-03-25T19:20:53.089 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:53.089+0000 7f3e2e7fc640 1 -- 172.21.15.83:0/965218504 <== mon.1 v1:172.21.15.103:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3e3404f180 con 0x7f3e3806c380 2024-03-25T19:20:53.089 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:53.089+0000 7f3e2e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:53.090 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:53.089+0000 7f3e2e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:20:53.313 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:53.312+0000 7f3e4020f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:53.313 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:53.312+0000 7f3e4020f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:53.313 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:53.312+0000 7f3e4020f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:20:53.313 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:53.312+0000 7f3e4020f640 1 -- 172.21.15.83:0/965218504 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3e381c5040 con 0x7f3e3806c380 2024-03-25T19:20:53.404 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:53.403+0000 7f3e2e7fc640 1 -- 172.21.15.83:0/965218504 <== mon.1 v1:172.21.15.103:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7f3e340510b0 con 0x7f3e3806c380 2024-03-25T19:20:53.404 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:53.403+0000 7f3e2e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:53.404 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:53.403+0000 7f3e2e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:20:53.404 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:20:53.404 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:20:53.407 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:53.406+0000 7f3e4020f640 10 librados: watch_flush enter 2024-03-25T19:20:53.407 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:53.406+0000 7f3e4020f640 10 librados: watch_flush exit 2024-03-25T19:20:53.407 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:53.406+0000 7f3e4020f640 20 client.4276.objecter shutdown clearing up homeless session... 2024-03-25T19:20:53.407 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:53.406+0000 7f3e4020f640 10 client.4276.objecter successfully canceled tick 2024-03-25T19:20:53.407 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:53.406+0000 7f3e4020f640 1 -- 172.21.15.83:0/965218504 >> v1:172.21.15.103:6800/345072327 conn(0x7f3e1c06df70 legacy=0x7f3e1c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:53.407 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:53.406+0000 7f3e4020f640 10 monclient: shutdown 2024-03-25T19:20:53.407 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:53.406+0000 7f3e4020f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:53.408 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:53.406+0000 7f3e4020f640 1 -- 172.21.15.83:0/965218504 >> v1:172.21.15.103:6789/0 conn(0x7f3e3806c380 legacy=0x7f3e381d7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:53.408 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:53.406+0000 7f3e4020f640 1 -- 172.21.15.83:0/965218504 shutdown_connections 2024-03-25T19:20:53.408 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:53.406+0000 7f3e4020f640 1 -- 172.21.15.83:0/965218504 >> 172.21.15.83:0/965218504 conn(0x7f3e380b5d10 msgr2=0x7f3e381cd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:53.408 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:53.406+0000 7f3e4020f640 1 -- 172.21.15.83:0/965218504 shutdown_connections 2024-03-25T19:20:53.408 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:53.406+0000 7f3e4020f640 1 -- 172.21.15.83:0/965218504 wait complete. 2024-03-25T19:20:53.408 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:20:53.406+0000 7f3e4020f640 1 librados: shutdown 2024-03-25T19:20:53.417 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:21:00.223 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:21:00.340 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.340+0000 7ffb405cf640 10 monclient: get_monmap_and_config 2024-03-25T19:21:00.340 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.340+0000 7ffb405cf640 10 monclient: build_initial_monmap 2024-03-25T19:21:00.340 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.340+0000 7ffb405cf640 10 monclient: monmap: 2024-03-25T19:21:00.340 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:21:00.340 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:21:00.340 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:21:00.341303+0000 2024-03-25T19:21:00.340 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:21:00.341303+0000 2024-03-25T19:21:00.340 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:21:00.341 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:21:00.341 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:21:00.341 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:21:00.341 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:21:00.341 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:00.342 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.341+0000 7ffb405cf640 1 Processor -- start 2024-03-25T19:21:00.342 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.341+0000 7ffb405cf640 1 -- start start 2024-03-25T19:21:00.342 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.341+0000 7ffb405cf640 10 monclient: init 2024-03-25T19:21:00.342 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.341+0000 7ffb405cf640 10 monclient: _reopen_session rank -1 2024-03-25T19:21:00.342 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.341+0000 7ffb405cf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:21:00.342 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.341+0000 7ffb405cf640 10 monclient(hunting): picked mon.noname-c con 0x7ffb380a98f0 addr v1:172.21.15.103:6789/0 2024-03-25T19:21:00.342 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.341+0000 7ffb405cf640 10 monclient(hunting): picked mon.noname-a con 0x7ffb3813f710 addr v1:172.21.15.83:6789/0 2024-03-25T19:21:00.342 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.341+0000 7ffb405cf640 10 monclient(hunting): picked mon.noname-b con 0x7ffb381430c0 addr v1:172.21.15.83:6790/0 2024-03-25T19:21:00.343 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.341+0000 7ffb405cf640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb38142520 con 0x7ffb3813f710 2024-03-25T19:21:00.343 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.342+0000 7ffb405cf640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb380b28a0 con 0x7ffb380a98f0 2024-03-25T19:21:00.343 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.342+0000 7ffb405cf640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb380b2a30 con 0x7ffb381430c0 2024-03-25T19:21:00.343 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.342+0000 7ffb405cf640 10 monclient(hunting): _renew_subs 2024-03-25T19:21:00.343 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.342+0000 7ffb405cf640 10 monclient(hunting): authenticate will time out at 6117.613781s 2024-03-25T19:21:00.349 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.349+0000 7ffb3eb6c640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7ffb381430c0 0x7ffb3814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:45762/0 (socket says 172.21.15.83:45762) 2024-03-25T19:21:00.349 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.349+0000 7ffb3eb6c640 1 -- 172.21.15.83:0/2536509217 learned_addr learned my addr 172.21.15.83:0/2536509217 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:21:00.370 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.371+0000 7ffb3d369640 1 -- 172.21.15.83:0/2536509217 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 770874594 0 0) 0x7ffb380b2a30 con 0x7ffb381430c0 2024-03-25T19:21:00.370 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.371+0000 7ffb3d369640 10 monclient(hunting): _negotiate 2024-03-25T19:21:00.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.371+0000 7ffb3d369640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:00.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.371+0000 7ffb3d369640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:00.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.371+0000 7ffb3d369640 10 monclient(hunting): my global_id is 4283 2024-03-25T19:21:00.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.371+0000 7ffb3d369640 1 -- 172.21.15.83:0/2536509217 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb24003520 con 0x7ffb381430c0 2024-03-25T19:21:00.379 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.379+0000 7ffb3d369640 1 -- 172.21.15.83:0/2536509217 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4025302719 0 0) 0x7ffb380b28a0 con 0x7ffb380a98f0 2024-03-25T19:21:00.379 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.379+0000 7ffb3d369640 10 monclient(hunting): _negotiate 2024-03-25T19:21:00.379 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.379+0000 7ffb3d369640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:00.379 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.379+0000 7ffb3d369640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:00.379 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.379+0000 7ffb3d369640 10 monclient(hunting): my global_id is 4279 2024-03-25T19:21:00.379 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.379+0000 7ffb3d369640 1 -- 172.21.15.83:0/2536509217 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb380b2a30 con 0x7ffb380a98f0 2024-03-25T19:21:00.381 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.381+0000 7ffb3d369640 1 -- 172.21.15.83:0/2536509217 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1041077366 0 0) 0x7ffb38142520 con 0x7ffb3813f710 2024-03-25T19:21:00.381 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.381+0000 7ffb3d369640 10 monclient(hunting): _negotiate 2024-03-25T19:21:00.381 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.381+0000 7ffb3d369640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:00.381 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.381+0000 7ffb3d369640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:00.381 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.381+0000 7ffb3d369640 10 monclient(hunting): my global_id is 4272 2024-03-25T19:21:00.381 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.381+0000 7ffb3d369640 1 -- 172.21.15.83:0/2536509217 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb380b28a0 con 0x7ffb3813f710 2024-03-25T19:21:00.388 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.388+0000 7ffb3d369640 1 -- 172.21.15.83:0/2536509217 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2781261886 0 0) 0x7ffb24003520 con 0x7ffb381430c0 2024-03-25T19:21:00.388 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.388+0000 7ffb3d369640 1 -- 172.21.15.83:0/2536509217 >> v1:172.21.15.103:6789/0 conn(0x7ffb380a98f0 legacy=0x7ffb380a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:00.389 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.388+0000 7ffb3d369640 1 -- 172.21.15.83:0/2536509217 >> v1:172.21.15.83:6789/0 conn(0x7ffb3813f710 legacy=0x7ffb381429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:00.389 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.388+0000 7ffb3d369640 10 monclient: _finish_hunting 0 2024-03-25T19:21:00.389 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.388+0000 7ffb3d369640 1 monclient: found mon.noname-b 2024-03-25T19:21:00.389 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.388+0000 7ffb3d369640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:21:00.389 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.388+0000 7ffb3d369640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.83:6790/0 2024-03-25T19:21:00.389 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.388+0000 7ffb3d369640 1 -- 172.21.15.83:0/2536509217 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb380b44c0 con 0x7ffb381430c0 2024-03-25T19:21:00.389 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.388+0000 7ffb3d369640 10 monclient: _finish_auth 0 2024-03-25T19:21:00.389 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.388+0000 7ffb3d369640 10 monclient: _check_auth_tickets 2024-03-25T19:21:00.389 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.388+0000 7ffb3d369640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:21:00.389 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.388+0000 7ffb3d369640 1 -- 172.21.15.83:0/2536509217 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7ffb30001a10 con 0x7ffb381430c0 2024-03-25T19:21:00.389 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.388+0000 7ffb3d369640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:00.389 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.388+0000 7ffb3d369640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:21:00.390 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.388+0000 7ffb3d369640 10 monclient: dump: 2024-03-25T19:21:00.390 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:21:00.390 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:21:00.390 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:00.390 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:00.390 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:00.390 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:21:00.390 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:21:00.390 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:21:00.390 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:21:00.390 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:00.391 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.388+0000 7ffb405cf640 5 monclient: authenticate success, global_id 4283 2024-03-25T19:21:00.391 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.388+0000 7ffb405cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:21:00.393 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.393+0000 7ffb3d369640 1 -- 172.21.15.83:0/2536509217 <== mon.2 v1:172.21.15.83:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb30001b70 con 0x7ffb381430c0 2024-03-25T19:21:00.393 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.393+0000 7ffb3d369640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:21:00.393 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.393+0000 7ffb3d369640 1 -- 172.21.15.83:0/2536509217 <== mon.2 v1:172.21.15.83:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7ffb30001f80 con 0x7ffb381430c0 2024-03-25T19:21:00.393 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.393+0000 7ffb3d369640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:00.393 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.393+0000 7ffb3d369640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:21:00.393 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.393+0000 7ffb3d369640 10 monclient: dump: 2024-03-25T19:21:00.393 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:21:00.393 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:21:00.394 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:00.394 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:00.394 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:00.394 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:21:00.394 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:21:00.394 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:21:00.394 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:21:00.394 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:00.394 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.393+0000 7ffb405cf640 10 monclient: get_monmap_and_config success 2024-03-25T19:21:00.394 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.393+0000 7ffb405cf640 4 set_mon_vals no callback set 2024-03-25T19:21:00.397 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.394+0000 7ffb405cf640 10 monclient: shutdown 2024-03-25T19:21:00.397 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.394+0000 7ffb405cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:21:00.397 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.394+0000 7ffb405cf640 1 -- 172.21.15.83:0/2536509217 >> v1:172.21.15.83:6790/0 conn(0x7ffb381430c0 legacy=0x7ffb3814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:00.397 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.394+0000 7ffb405cf640 1 -- 172.21.15.83:0/2536509217 shutdown_connections 2024-03-25T19:21:00.397 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.394+0000 7ffb405cf640 1 -- 172.21.15.83:0/2536509217 >> 172.21.15.83:0/2536509217 conn(0x7ffb380b7ea0 msgr2=0x7ffb380b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:21:00.397 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.394+0000 7ffb405cf640 1 -- 172.21.15.83:0/2536509217 shutdown_connections 2024-03-25T19:21:00.397 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.394+0000 7ffb405cf640 1 -- 172.21.15.83:0/2536509217 wait complete. 2024-03-25T19:21:00.397 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.395+0000 7ffb405cf640 10 monclient: build_initial_monmap 2024-03-25T19:21:00.397 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.395+0000 7ffb405cf640 10 monclient: monmap: 2024-03-25T19:21:00.398 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:21:00.398 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:21:00.398 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:21:00.398 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:21:00.398 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:21:00.398 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:21:00.398 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:21:00.398 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:21:00.398 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:21:00.398 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:00.398 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.396+0000 7ffb405cf640 1 librados: starting msgr at 2024-03-25T19:21:00.398 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.396+0000 7ffb405cf640 1 librados: starting objecter 2024-03-25T19:21:00.399 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.396+0000 7ffb405cf640 1 Processor -- start 2024-03-25T19:21:00.399 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.396+0000 7ffb405cf640 1 -- start start 2024-03-25T19:21:00.399 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.396+0000 7ffb405cf640 1 librados: setting wanted keys 2024-03-25T19:21:00.399 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.396+0000 7ffb405cf640 1 librados: calling monclient init 2024-03-25T19:21:00.399 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.396+0000 7ffb405cf640 10 monclient: init 2024-03-25T19:21:00.399 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.397+0000 7ffb405cf640 10 monclient: _reopen_session rank -1 2024-03-25T19:21:00.399 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.397+0000 7ffb405cf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:21:00.399 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.397+0000 7ffb405cf640 10 monclient(hunting): picked mon.noname-b con 0x7ffb380a98f0 addr v1:172.21.15.103:6789/0 2024-03-25T19:21:00.399 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.397+0000 7ffb405cf640 10 monclient(hunting): picked mon.noname-c con 0x7ffb3813f710 addr v1:172.21.15.83:6790/0 2024-03-25T19:21:00.399 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.397+0000 7ffb405cf640 10 monclient(hunting): picked mon.noname-a con 0x7ffb381430c0 addr v1:172.21.15.83:6789/0 2024-03-25T19:21:00.399 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.397+0000 7ffb405cf640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb3813c9b0 con 0x7ffb381430c0 2024-03-25T19:21:00.400 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.397+0000 7ffb405cf640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb381397c0 con 0x7ffb380a98f0 2024-03-25T19:21:00.400 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.397+0000 7ffb405cf640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb38139950 con 0x7ffb3813f710 2024-03-25T19:21:00.400 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.397+0000 7ffb405cf640 10 monclient(hunting): _renew_subs 2024-03-25T19:21:00.400 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.397+0000 7ffb405cf640 10 monclient(hunting): authenticate will time out at 6117.668883s 2024-03-25T19:21:00.410 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.410+0000 7ffb3eb6c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ffb381430c0 0x7ffb381e2f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:54840/0 (socket says 172.21.15.83:54840) 2024-03-25T19:21:00.410 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.410+0000 7ffb3eb6c640 1 -- 172.21.15.83:0/1317036148 learned_addr learned my addr 172.21.15.83:0/1317036148 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:21:00.413 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.413+0000 7ffb2effd640 10 client.?.objecter ms_handle_connect 0x7ffb381430c0 2024-03-25T19:21:00.413 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.413+0000 7ffb2effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:00.415 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.415+0000 7ffb2effd640 10 client.?.objecter ms_handle_connect 0x7ffb380a98f0 2024-03-25T19:21:00.415 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.415+0000 7ffb2effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:00.418 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.418+0000 7ffb2effd640 1 -- 172.21.15.83:0/1317036148 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 421814388 0 0) 0x7ffb3813c9b0 con 0x7ffb381430c0 2024-03-25T19:21:00.418 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.418+0000 7ffb2effd640 10 monclient(hunting): _negotiate 2024-03-25T19:21:00.418 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.418+0000 7ffb2effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:00.418 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.418+0000 7ffb2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:00.418 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.418+0000 7ffb2effd640 10 monclient(hunting): my global_id is 4275 2024-03-25T19:21:00.418 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.418+0000 7ffb2effd640 1 -- 172.21.15.83:0/1317036148 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb1c004eb0 con 0x7ffb381430c0 2024-03-25T19:21:00.421 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.422+0000 7ffb2effd640 10 client.?.objecter ms_handle_connect 0x7ffb3813f710 2024-03-25T19:21:00.421 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.422+0000 7ffb2effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:00.422 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.422+0000 7ffb2effd640 1 -- 172.21.15.83:0/1317036148 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1554512034 0 0) 0x7ffb381397c0 con 0x7ffb380a98f0 2024-03-25T19:21:00.422 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.422+0000 7ffb2effd640 10 monclient(hunting): _negotiate 2024-03-25T19:21:00.422 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.422+0000 7ffb2effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:00.422 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.422+0000 7ffb2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:00.422 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.422+0000 7ffb2effd640 10 monclient(hunting): my global_id is 4282 2024-03-25T19:21:00.422 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.422+0000 7ffb2effd640 1 -- 172.21.15.83:0/1317036148 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb3813c9b0 con 0x7ffb380a98f0 2024-03-25T19:21:00.425 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.426+0000 7ffb2effd640 1 -- 172.21.15.83:0/1317036148 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2512278890 0 0) 0x7ffb38139950 con 0x7ffb3813f710 2024-03-25T19:21:00.425 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.426+0000 7ffb2effd640 10 monclient(hunting): _negotiate 2024-03-25T19:21:00.426 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.426+0000 7ffb2effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:00.426 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.426+0000 7ffb2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:00.426 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.426+0000 7ffb2effd640 10 monclient(hunting): my global_id is 4286 2024-03-25T19:21:00.426 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.426+0000 7ffb2effd640 1 -- 172.21.15.83:0/1317036148 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb381397c0 con 0x7ffb3813f710 2024-03-25T19:21:00.429 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.429+0000 7ffb2effd640 1 -- 172.21.15.83:0/1317036148 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3496441372 0 0) 0x7ffb1c004eb0 con 0x7ffb381430c0 2024-03-25T19:21:00.429 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.430+0000 7ffb2effd640 1 -- 172.21.15.83:0/1317036148 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb38139950 con 0x7ffb381430c0 2024-03-25T19:21:00.430 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.430+0000 7ffb2effd640 1 -- 172.21.15.83:0/1317036148 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7ffb30001ec0 con 0x7ffb381430c0 2024-03-25T19:21:00.430 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.430+0000 7ffb2effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:00.430 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.430+0000 7ffb2effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:21:00.430 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.430+0000 7ffb2effd640 10 monclient(hunting): dump: 2024-03-25T19:21:00.430 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:21:00.430 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:21:00.430 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:00.430 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:00.430 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:00.430 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:21:00.430 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:21:00.430 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:21:00.431 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:21:00.431 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:00.431 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.430+0000 7ffb2effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:21:00.434 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.435+0000 7ffb2effd640 1 -- 172.21.15.83:0/1317036148 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1316988700 0 0) 0x7ffb3813c9b0 con 0x7ffb380a98f0 2024-03-25T19:21:00.435 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.435+0000 7ffb2effd640 1 -- 172.21.15.83:0/1317036148 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb1c004eb0 con 0x7ffb380a98f0 2024-03-25T19:21:00.435 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.435+0000 7ffb2effd640 1 -- 172.21.15.83:0/1317036148 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7ffb34001970 con 0x7ffb380a98f0 2024-03-25T19:21:00.435 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.435+0000 7ffb2effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:00.435 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.435+0000 7ffb2effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:21:00.435 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.435+0000 7ffb2effd640 10 monclient(hunting): dump: 2024-03-25T19:21:00.435 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:21:00.435 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:21:00.435 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:00.435 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:00.435 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:00.435 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:21:00.435 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:21:00.436 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:21:00.436 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:21:00.436 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:00.439 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.439+0000 7ffb2effd640 1 -- 172.21.15.83:0/1317036148 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2801538834 0 0) 0x7ffb38139950 con 0x7ffb381430c0 2024-03-25T19:21:00.439 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.439+0000 7ffb2effd640 1 -- 172.21.15.83:0/1317036148 >> v1:172.21.15.83:6790/0 conn(0x7ffb3813f710 legacy=0x7ffb381dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:00.439 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.439+0000 7ffb2effd640 1 -- 172.21.15.83:0/1317036148 >> v1:172.21.15.103:6789/0 conn(0x7ffb380a98f0 legacy=0x7ffb38130f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:00.439 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.439+0000 7ffb2effd640 10 monclient: _finish_hunting 0 2024-03-25T19:21:00.439 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.439+0000 7ffb2effd640 1 monclient: found mon.a 2024-03-25T19:21:00.439 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.439+0000 7ffb2effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:21:00.439 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.439+0000 7ffb2effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:21:00.440 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.439+0000 7ffb2effd640 1 -- 172.21.15.83:0/1317036148 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb38131680 con 0x7ffb381430c0 2024-03-25T19:21:00.440 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.439+0000 7ffb2effd640 10 monclient: _finish_auth 0 2024-03-25T19:21:00.440 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.439+0000 7ffb2effd640 10 monclient: _check_auth_tickets 2024-03-25T19:21:00.440 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.439+0000 7ffb2effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:21:00.440 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.439+0000 7ffb405cf640 5 monclient: authenticate success, global_id 4275 2024-03-25T19:21:00.440 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.439+0000 7ffb405cf640 10 monclient: _renew_subs 2024-03-25T19:21:00.440 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.439+0000 7ffb405cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:21:00.440 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.439+0000 7ffb405cf640 1 -- 172.21.15.83:0/1317036148 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb38131950 con 0x7ffb381430c0 2024-03-25T19:21:00.440 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.439+0000 7ffb405cf640 10 client.4275.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:21:00.440 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.439+0000 7ffb405cf640 10 monclient: _renew_subs 2024-03-25T19:21:00.440 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.439+0000 7ffb405cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:21:00.440 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.439+0000 7ffb405cf640 1 -- 172.21.15.83:0/1317036148 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb38135a30 con 0x7ffb381430c0 2024-03-25T19:21:00.440 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.439+0000 7ffb405cf640 1 librados: init done 2024-03-25T19:21:00.441 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.440+0000 7ffb405cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:00.441 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.440+0000 7ffb405cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:00.441 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.440+0000 7ffb405cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:21:00.441 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.440+0000 7ffb405cf640 1 -- 172.21.15.83:0/1317036148 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb3813c810 con 0x7ffb381430c0 2024-03-25T19:21:00.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.729+0000 7ffb2effd640 1 -- 172.21.15.83:0/1317036148 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb30004e10 con 0x7ffb381430c0 2024-03-25T19:21:00.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.729+0000 7ffb2effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:21:00.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.729+0000 7ffb2effd640 1 -- 172.21.15.83:0/1317036148 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7ffb300015d0 con 0x7ffb381430c0 2024-03-25T19:21:00.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.729+0000 7ffb2effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:00.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.729+0000 7ffb2effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:21:00.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.729+0000 7ffb2effd640 10 monclient: dump: 2024-03-25T19:21:00.729 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:21:00.729 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:21:00.730 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:00.730 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:00.730 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:00.730 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:21:00.730 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:21:00.730 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:21:00.730 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:21:00.730 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:00.730 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.729+0000 7ffb3cb68640 4 set_mon_vals no callback set 2024-03-25T19:21:00.731 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.730+0000 7ffb2effd640 1 -- 172.21.15.83:0/1317036148 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7ffb3001adf0 con 0x7ffb381430c0 2024-03-25T19:21:00.731 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.731+0000 7ffb3e36b640 10 monclient: get_auth_request con 0x7ffb1c06df20 auth_method 0 2024-03-25T19:21:00.731 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.731+0000 7ffb2effd640 10 client.4275.objecter ms_handle_connect 0x7ffb1c06df20 2024-03-25T19:21:00.735 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.735+0000 7ffb2effd640 1 -- 172.21.15.83:0/1317036148 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7ffb30054a00 con 0x7ffb381430c0 2024-03-25T19:21:00.735 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.735+0000 7ffb2effd640 10 client.4275.objecter ms_dispatch 0x7ffb38000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:21:00.735 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.735+0000 7ffb2effd640 3 client.4275.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:21:00.735 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.735+0000 7ffb2effd640 3 client.4275.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:21:00.735 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.735+0000 7ffb2effd640 20 client.4275.objecter dump_active .. 0 homeless 2024-03-25T19:21:00.735 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.735+0000 7ffb2effd640 1 -- 172.21.15.83:0/1317036148 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffb30050510 con 0x7ffb381430c0 2024-03-25T19:21:00.735 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.735+0000 7ffb2effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:00.735 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.735+0000 7ffb2effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:21:00.993 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.993+0000 7ffb405cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:00.993 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.993+0000 7ffb405cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:00.993 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.993+0000 7ffb405cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:21:00.993 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:00.993+0000 7ffb405cf640 1 -- 172.21.15.83:0/1317036148 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb38136fc0 con 0x7ffb381430c0 2024-03-25T19:21:01.026 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:01.026+0000 7ffb2effd640 1 -- 172.21.15.83:0/1317036148 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7ffb30052440 con 0x7ffb381430c0 2024-03-25T19:21:01.026 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:01.026+0000 7ffb2effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:01.026 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:01.026+0000 7ffb2effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:21:01.026 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:21:01.026 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:21:01.029 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:01.029+0000 7ffb405cf640 10 librados: watch_flush enter 2024-03-25T19:21:01.029 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:01.029+0000 7ffb405cf640 10 librados: watch_flush exit 2024-03-25T19:21:01.029 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:01.029+0000 7ffb405cf640 20 client.4275.objecter shutdown clearing up homeless session... 2024-03-25T19:21:01.030 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:01.029+0000 7ffb405cf640 10 client.4275.objecter successfully canceled tick 2024-03-25T19:21:01.030 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:01.029+0000 7ffb405cf640 1 -- 172.21.15.83:0/1317036148 >> v1:172.21.15.103:6800/345072327 conn(0x7ffb1c06df20 legacy=0x7ffb1c0703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:01.030 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:01.029+0000 7ffb405cf640 10 monclient: shutdown 2024-03-25T19:21:01.030 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:01.029+0000 7ffb405cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:21:01.030 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:01.029+0000 7ffb405cf640 1 -- 172.21.15.83:0/1317036148 >> v1:172.21.15.83:6789/0 conn(0x7ffb381430c0 legacy=0x7ffb381e2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:01.030 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:01.029+0000 7ffb405cf640 1 -- 172.21.15.83:0/1317036148 shutdown_connections 2024-03-25T19:21:01.030 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:01.029+0000 7ffb405cf640 1 -- 172.21.15.83:0/1317036148 >> 172.21.15.83:0/1317036148 conn(0x7ffb380b7ea0 msgr2=0x7ffb3814ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:21:01.030 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:01.029+0000 7ffb405cf640 1 -- 172.21.15.83:0/1317036148 shutdown_connections 2024-03-25T19:21:01.030 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:01.029+0000 7ffb405cf640 1 -- 172.21.15.83:0/1317036148 wait complete. 2024-03-25T19:21:01.030 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:01.029+0000 7ffb405cf640 1 librados: shutdown 2024-03-25T19:21:01.041 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:21:02.293 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:01:44.221 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:02.305 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:01:44.233 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:02.347 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:01:44.257 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:02.355 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:01:44.265 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:02.361 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:01:44.289 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:02.365 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:01:44.292 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:02.370 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:01:44.297 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:02.394 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:01:44.304 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:02.419 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:01:44.347 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:07.848 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:21:07.966 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:07.966+0000 7f3653596640 10 monclient: get_monmap_and_config 2024-03-25T19:21:07.966 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:07.966+0000 7f3653596640 10 monclient: build_initial_monmap 2024-03-25T19:21:07.966 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:07.966+0000 7f3653596640 10 monclient: monmap: 2024-03-25T19:21:07.966 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:21:07.966 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:21:07.966 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:21:07.967410+0000 2024-03-25T19:21:07.966 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:21:07.967410+0000 2024-03-25T19:21:07.966 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:21:07.966 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:21:07.967 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:21:07.967 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:21:07.967 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:21:07.967 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:07.968 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:07.967+0000 7f3653596640 1 Processor -- start 2024-03-25T19:21:07.968 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:07.967+0000 7f3653596640 1 -- start start 2024-03-25T19:21:07.968 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:07.967+0000 7f3653596640 10 monclient: init 2024-03-25T19:21:07.968 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:07.968+0000 7f3653596640 10 monclient: _reopen_session rank -1 2024-03-25T19:21:07.968 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:07.968+0000 7f3653596640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:21:07.968 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:07.968+0000 7f3653596640 10 monclient(hunting): picked mon.noname-c con 0x7f364c0a98f0 addr v1:172.21.15.103:6789/0 2024-03-25T19:21:07.968 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:07.968+0000 7f3653596640 10 monclient(hunting): picked mon.noname-b con 0x7f364c13f710 addr v1:172.21.15.83:6790/0 2024-03-25T19:21:07.969 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:07.968+0000 7f3653596640 10 monclient(hunting): picked mon.noname-a con 0x7f364c1430c0 addr v1:172.21.15.83:6789/0 2024-03-25T19:21:07.969 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:07.968+0000 7f3653596640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f364c142520 con 0x7f364c1430c0 2024-03-25T19:21:07.969 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:07.968+0000 7f3653596640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f364c0b28a0 con 0x7f364c0a98f0 2024-03-25T19:21:07.969 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:07.968+0000 7f3653596640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f364c0b2a30 con 0x7f364c13f710 2024-03-25T19:21:07.969 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:07.968+0000 7f3653596640 10 monclient(hunting): _renew_subs 2024-03-25T19:21:07.969 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:07.968+0000 7f3653596640 10 monclient(hunting): authenticate will time out at 6125.240162s 2024-03-25T19:21:07.982 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:07.983+0000 7f3651332640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f364c0a98f0 0x7f364c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:53480/0 (socket says 172.21.15.83:53480) 2024-03-25T19:21:07.982 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:07.983+0000 7f3651332640 1 -- 172.21.15.83:0/3206278056 learned_addr learned my addr 172.21.15.83:0/3206278056 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:21:07.990 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:07.991+0000 7f3643fff640 1 -- 172.21.15.83:0/3206278056 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1398208541 0 0) 0x7f364c0b28a0 con 0x7f364c0a98f0 2024-03-25T19:21:07.990 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:07.991+0000 7f3643fff640 10 monclient(hunting): _negotiate 2024-03-25T19:21:07.991 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:07.991+0000 7f3643fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:07.991 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:07.991+0000 7f3643fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:07.991 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:07.991+0000 7f3643fff640 10 monclient(hunting): my global_id is 4285 2024-03-25T19:21:07.991 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:07.991+0000 7f3643fff640 1 -- 172.21.15.83:0/3206278056 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3638003520 con 0x7f364c0a98f0 2024-03-25T19:21:08.004 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.004+0000 7f3643fff640 1 -- 172.21.15.83:0/3206278056 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1020910827 0 0) 0x7f364c0b2a30 con 0x7f364c13f710 2024-03-25T19:21:08.004 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.004+0000 7f3643fff640 10 monclient(hunting): _negotiate 2024-03-25T19:21:08.004 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.004+0000 7f3643fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:08.004 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.004+0000 7f3643fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:08.005 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.004+0000 7f3643fff640 10 monclient(hunting): my global_id is 4289 2024-03-25T19:21:08.005 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.005+0000 7f3643fff640 1 -- 172.21.15.83:0/3206278056 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f364c0b28a0 con 0x7f364c13f710 2024-03-25T19:21:08.006 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.005+0000 7f3643fff640 1 -- 172.21.15.83:0/3206278056 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3197014011 0 0) 0x7f3638003520 con 0x7f364c0a98f0 2024-03-25T19:21:08.006 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.005+0000 7f3643fff640 1 -- 172.21.15.83:0/3206278056 >> v1:172.21.15.83:6790/0 conn(0x7f364c13f710 legacy=0x7f364c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:08.006 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.005+0000 7f3643fff640 1 -- 172.21.15.83:0/3206278056 >> v1:172.21.15.83:6789/0 conn(0x7f364c1430c0 legacy=0x7f364c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:08.006 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.006+0000 7f3643fff640 10 monclient: _finish_hunting 0 2024-03-25T19:21:08.006 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.006+0000 7f3643fff640 1 monclient: found mon.noname-c 2024-03-25T19:21:08.006 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.006+0000 7f3643fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:21:08.006 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.006+0000 7f3643fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.103:6789/0 2024-03-25T19:21:08.006 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.006+0000 7f3643fff640 1 -- 172.21.15.83:0/3206278056 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f364c0b44c0 con 0x7f364c0a98f0 2024-03-25T19:21:08.006 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.006+0000 7f3643fff640 10 monclient: _finish_auth 0 2024-03-25T19:21:08.006 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.006+0000 7f3643fff640 10 monclient: _check_auth_tickets 2024-03-25T19:21:08.006 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.006+0000 7f3643fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:21:08.006 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.006+0000 7f3643fff640 1 -- 172.21.15.83:0/3206278056 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f3648001690 con 0x7f364c0a98f0 2024-03-25T19:21:08.006 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.006+0000 7f3643fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:08.007 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.006+0000 7f3643fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:21:08.007 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.006+0000 7f3643fff640 10 monclient: dump: 2024-03-25T19:21:08.007 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:21:08.007 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:21:08.007 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:08.007 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:08.007 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:08.007 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:21:08.007 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:21:08.007 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:21:08.007 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:21:08.007 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:08.007 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.006+0000 7f3653596640 5 monclient: authenticate success, global_id 4285 2024-03-25T19:21:08.008 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.006+0000 7f3653596640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:21:08.025 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.025+0000 7f3643fff640 1 -- 172.21.15.83:0/3206278056 <== mon.1 v1:172.21.15.103:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36480017f0 con 0x7f364c0a98f0 2024-03-25T19:21:08.025 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.025+0000 7f3643fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:21:08.025 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.025+0000 7f3643fff640 1 -- 172.21.15.83:0/3206278056 <== mon.1 v1:172.21.15.103:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f3648001c00 con 0x7f364c0a98f0 2024-03-25T19:21:08.025 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.025+0000 7f3653596640 10 monclient: get_monmap_and_config success 2024-03-25T19:21:08.025 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.025+0000 7f3653596640 4 set_mon_vals no callback set 2024-03-25T19:21:08.025 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.025+0000 7f3643fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:08.025 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.025+0000 7f3643fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:21:08.025 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.025+0000 7f3643fff640 10 monclient: dump: 2024-03-25T19:21:08.025 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:21:08.025 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:21:08.025 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:08.025 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:08.026 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:08.026 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:21:08.026 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:21:08.026 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:21:08.026 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:21:08.026 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:08.028 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.026+0000 7f3653596640 10 monclient: shutdown 2024-03-25T19:21:08.028 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.026+0000 7f3653596640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:21:08.028 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.026+0000 7f3653596640 1 -- 172.21.15.83:0/3206278056 >> v1:172.21.15.103:6789/0 conn(0x7f364c0a98f0 legacy=0x7f364c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:08.028 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.026+0000 7f3653596640 1 -- 172.21.15.83:0/3206278056 shutdown_connections 2024-03-25T19:21:08.028 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.026+0000 7f3653596640 1 -- 172.21.15.83:0/3206278056 >> 172.21.15.83:0/3206278056 conn(0x7f364c0b7ea0 msgr2=0x7f364c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:21:08.029 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.026+0000 7f3653596640 1 -- 172.21.15.83:0/3206278056 shutdown_connections 2024-03-25T19:21:08.029 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.026+0000 7f3653596640 1 -- 172.21.15.83:0/3206278056 wait complete. 2024-03-25T19:21:08.029 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.027+0000 7f3653596640 10 monclient: build_initial_monmap 2024-03-25T19:21:08.029 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.027+0000 7f3653596640 10 monclient: monmap: 2024-03-25T19:21:08.029 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:21:08.029 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:21:08.029 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:21:08.029 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:21:08.029 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:21:08.029 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:21:08.029 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:21:08.029 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:21:08.029 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:21:08.029 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:08.029 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.027+0000 7f3653596640 1 librados: starting msgr at 2024-03-25T19:21:08.030 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.027+0000 7f3653596640 1 librados: starting objecter 2024-03-25T19:21:08.030 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.027+0000 7f3653596640 1 Processor -- start 2024-03-25T19:21:08.030 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.027+0000 7f3653596640 1 -- start start 2024-03-25T19:21:08.030 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.027+0000 7f3653596640 1 librados: setting wanted keys 2024-03-25T19:21:08.030 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.027+0000 7f3653596640 1 librados: calling monclient init 2024-03-25T19:21:08.030 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.027+0000 7f3653596640 10 monclient: init 2024-03-25T19:21:08.030 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.028+0000 7f3653596640 10 monclient: _reopen_session rank -1 2024-03-25T19:21:08.030 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.028+0000 7f3653596640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:21:08.030 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.028+0000 7f3653596640 10 monclient(hunting): picked mon.noname-b con 0x7f364c0a98f0 addr v1:172.21.15.103:6789/0 2024-03-25T19:21:08.030 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.028+0000 7f3653596640 10 monclient(hunting): picked mon.noname-a con 0x7f364c13f710 addr v1:172.21.15.83:6789/0 2024-03-25T19:21:08.030 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.028+0000 7f3653596640 10 monclient(hunting): picked mon.noname-c con 0x7f364c1430c0 addr v1:172.21.15.83:6790/0 2024-03-25T19:21:08.030 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.028+0000 7f3653596640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f364c135730 con 0x7f364c13f710 2024-03-25T19:21:08.030 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.028+0000 7f3653596640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f364c1358c0 con 0x7f364c0a98f0 2024-03-25T19:21:08.030 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.028+0000 7f3653596640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f364c135a50 con 0x7f364c1430c0 2024-03-25T19:21:08.030 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.028+0000 7f3653596640 10 monclient(hunting): _renew_subs 2024-03-25T19:21:08.031 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.028+0000 7f3653596640 10 monclient(hunting): authenticate will time out at 6125.300100s 2024-03-25T19:21:08.037 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.037+0000 7f3650b31640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f364c13f710 0x7f364c1d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:45152/0 (socket says 172.21.15.83:45152) 2024-03-25T19:21:08.037 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.037+0000 7f3650b31640 1 -- 172.21.15.83:0/2660917450 learned_addr learned my addr 172.21.15.83:0/2660917450 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:21:08.038 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.038+0000 7f3641ffb640 10 client.?.objecter ms_handle_connect 0x7f364c13f710 2024-03-25T19:21:08.038 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.038+0000 7f3641ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:08.042 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.043+0000 7f3641ffb640 10 client.?.objecter ms_handle_connect 0x7f364c0a98f0 2024-03-25T19:21:08.042 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.043+0000 7f3641ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:08.043 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.043+0000 7f3641ffb640 10 client.?.objecter ms_handle_connect 0x7f364c1430c0 2024-03-25T19:21:08.043 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.043+0000 7f3641ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:08.051 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.051+0000 7f3641ffb640 1 -- 172.21.15.83:0/2660917450 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 890753307 0 0) 0x7f364c1358c0 con 0x7f364c0a98f0 2024-03-25T19:21:08.051 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.051+0000 7f3641ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:21:08.051 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.051+0000 7f3641ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:08.051 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.051+0000 7f3641ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:08.051 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.051+0000 7f3641ffb640 10 monclient(hunting): my global_id is 4288 2024-03-25T19:21:08.051 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.051+0000 7f3641ffb640 1 -- 172.21.15.83:0/2660917450 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3630004eb0 con 0x7f364c0a98f0 2024-03-25T19:21:08.053 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.053+0000 7f3641ffb640 1 -- 172.21.15.83:0/2660917450 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4247702186 0 0) 0x7f364c135730 con 0x7f364c13f710 2024-03-25T19:21:08.053 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.053+0000 7f3641ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:21:08.053 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.053+0000 7f3641ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:08.053 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.053+0000 7f3641ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:08.053 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.053+0000 7f3641ffb640 10 monclient(hunting): my global_id is 4281 2024-03-25T19:21:08.053 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.053+0000 7f3641ffb640 1 -- 172.21.15.83:0/2660917450 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f364c1358c0 con 0x7f364c13f710 2024-03-25T19:21:08.057 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.057+0000 7f3641ffb640 1 -- 172.21.15.83:0/2660917450 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3380168889 0 0) 0x7f364c135a50 con 0x7f364c1430c0 2024-03-25T19:21:08.057 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.057+0000 7f3641ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:21:08.057 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.057+0000 7f3641ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:08.057 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.057+0000 7f3641ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:08.057 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.057+0000 7f3641ffb640 10 monclient(hunting): my global_id is 4292 2024-03-25T19:21:08.057 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.057+0000 7f3641ffb640 1 -- 172.21.15.83:0/2660917450 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f364c135730 con 0x7f364c1430c0 2024-03-25T19:21:08.069 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.069+0000 7f3641ffb640 1 -- 172.21.15.83:0/2660917450 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 403285578 0 0) 0x7f3630004eb0 con 0x7f364c0a98f0 2024-03-25T19:21:08.069 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.069+0000 7f3641ffb640 1 -- 172.21.15.83:0/2660917450 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f364c135a50 con 0x7f364c0a98f0 2024-03-25T19:21:08.069 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.069+0000 7f3641ffb640 1 -- 172.21.15.83:0/2660917450 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f3648001bc0 con 0x7f364c0a98f0 2024-03-25T19:21:08.069 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.069+0000 7f3641ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:08.069 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.069+0000 7f3641ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:21:08.069 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.069+0000 7f3641ffb640 10 monclient(hunting): dump: 2024-03-25T19:21:08.069 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:21:08.069 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:21:08.069 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:08.069 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:08.069 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:08.069 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:21:08.069 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:21:08.069 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:21:08.070 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:21:08.070 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:08.070 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.069+0000 7f3641ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:21:08.079 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.079+0000 7f3641ffb640 1 -- 172.21.15.83:0/2660917450 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1525207725 0 0) 0x7f364c1358c0 con 0x7f364c13f710 2024-03-25T19:21:08.079 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.079+0000 7f3641ffb640 1 -- 172.21.15.83:0/2660917450 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3630004eb0 con 0x7f364c13f710 2024-03-25T19:21:08.079 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.079+0000 7f3641ffb640 1 -- 172.21.15.83:0/2660917450 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f363c0019f0 con 0x7f364c13f710 2024-03-25T19:21:08.079 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.079+0000 7f3641ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:08.079 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.079+0000 7f3641ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:21:08.079 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.079+0000 7f3641ffb640 10 monclient(hunting): dump: 2024-03-25T19:21:08.079 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:21:08.079 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:21:08.080 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:08.080 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:08.080 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:08.080 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:21:08.080 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:21:08.080 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:21:08.080 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:21:08.080 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:08.081 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.080+0000 7f3641ffb640 1 -- 172.21.15.83:0/2660917450 <== mon.1 v1:172.21.15.103:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4188543766 0 0) 0x7f364c135a50 con 0x7f364c0a98f0 2024-03-25T19:21:08.081 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.080+0000 7f3641ffb640 1 -- 172.21.15.83:0/2660917450 >> v1:172.21.15.83:6790/0 conn(0x7f364c1430c0 legacy=0x7f364c1d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:08.081 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.081+0000 7f3641ffb640 1 -- 172.21.15.83:0/2660917450 >> v1:172.21.15.83:6789/0 conn(0x7f364c13f710 legacy=0x7f364c1d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:08.081 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.081+0000 7f3641ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:21:08.081 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.081+0000 7f3641ffb640 1 monclient: found mon.b 2024-03-25T19:21:08.081 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.081+0000 7f3641ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:21:08.082 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.081+0000 7f3641ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:21:08.082 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.081+0000 7f3641ffb640 1 -- 172.21.15.83:0/2660917450 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f364c1d5000 con 0x7f364c0a98f0 2024-03-25T19:21:08.082 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.081+0000 7f3641ffb640 10 monclient: _finish_auth 0 2024-03-25T19:21:08.082 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.081+0000 7f3641ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:21:08.082 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.081+0000 7f3641ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:21:08.082 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.081+0000 7f3653596640 5 monclient: authenticate success, global_id 4288 2024-03-25T19:21:08.082 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.081+0000 7f3653596640 10 monclient: _renew_subs 2024-03-25T19:21:08.082 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.081+0000 7f3653596640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:21:08.082 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.081+0000 7f3653596640 1 -- 172.21.15.83:0/2660917450 --> v1:172.21.15.103:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f364c1d52b0 con 0x7f364c0a98f0 2024-03-25T19:21:08.082 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.081+0000 7f3653596640 10 client.4288.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:21:08.082 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.081+0000 7f3653596640 10 monclient: _renew_subs 2024-03-25T19:21:08.082 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.081+0000 7f3653596640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:21:08.082 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.081+0000 7f3653596640 1 -- 172.21.15.83:0/2660917450 --> v1:172.21.15.103:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f364c13c900 con 0x7f364c0a98f0 2024-03-25T19:21:08.082 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.081+0000 7f3653596640 1 librados: init done 2024-03-25T19:21:08.083 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.082+0000 7f3653596640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:08.083 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.082+0000 7f3653596640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:08.083 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.082+0000 7f3653596640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:21:08.083 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.082+0000 7f3653596640 1 -- 172.21.15.83:0/2660917450 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f364c13cbb0 con 0x7f364c0a98f0 2024-03-25T19:21:08.360 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.361+0000 7f3641ffb640 1 -- 172.21.15.83:0/2660917450 <== mon.1 v1:172.21.15.103:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3648004a90 con 0x7f364c0a98f0 2024-03-25T19:21:08.361 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.361+0000 7f3641ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:21:08.361 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.361+0000 7f3641ffb640 1 -- 172.21.15.83:0/2660917450 <== mon.1 v1:172.21.15.103:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f3648004c00 con 0x7f364c0a98f0 2024-03-25T19:21:08.361 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.361+0000 7f3641ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:08.361 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.361+0000 7f3641ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:21:08.361 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.361+0000 7f3641ffb640 10 monclient: dump: 2024-03-25T19:21:08.361 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:21:08.361 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:21:08.361 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:08.361 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:08.361 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:08.361 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:21:08.361 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:21:08.362 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:21:08.362 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:21:08.362 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:08.362 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.361+0000 7f36437fe640 4 set_mon_vals no callback set 2024-03-25T19:21:08.362 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.362+0000 7f3641ffb640 1 -- 172.21.15.83:0/2660917450 <== mon.1 v1:172.21.15.103:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7f364801abd0 con 0x7f364c0a98f0 2024-03-25T19:21:08.362 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.362+0000 7f3650b31640 10 monclient: get_auth_request con 0x7f363006df70 auth_method 0 2024-03-25T19:21:08.363 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.363+0000 7f3641ffb640 10 client.4288.objecter ms_handle_connect 0x7f363006df70 2024-03-25T19:21:08.366 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.366+0000 7f3641ffb640 1 -- 172.21.15.83:0/2660917450 <== mon.1 v1:172.21.15.103:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7f36480547b0 con 0x7f364c0a98f0 2024-03-25T19:21:08.366 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.366+0000 7f3641ffb640 10 client.4288.objecter ms_dispatch 0x7f364c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:21:08.366 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.366+0000 7f3641ffb640 3 client.4288.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:21:08.367 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.366+0000 7f3641ffb640 3 client.4288.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:21:08.367 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.366+0000 7f3641ffb640 20 client.4288.objecter dump_active .. 0 homeless 2024-03-25T19:21:08.367 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.367+0000 7f3641ffb640 1 -- 172.21.15.83:0/2660917450 <== mon.1 v1:172.21.15.103:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f36480500f0 con 0x7f364c0a98f0 2024-03-25T19:21:08.367 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.367+0000 7f3641ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:08.367 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.367+0000 7f3641ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:21:08.599 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.599+0000 7f3653596640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:08.599 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.599+0000 7f3653596640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:08.599 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.599+0000 7f3653596640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:21:08.599 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.599+0000 7f3653596640 1 -- 172.21.15.83:0/2660917450 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f364c1371c0 con 0x7f364c0a98f0 2024-03-25T19:21:08.648 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.648+0000 7f3641ffb640 1 -- 172.21.15.83:0/2660917450 <== mon.1 v1:172.21.15.103:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7f36480521f0 con 0x7f364c0a98f0 2024-03-25T19:21:08.648 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.648+0000 7f3641ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:08.648 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.648+0000 7f3641ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:21:08.648 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:21:08.648 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:21:08.651 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.650+0000 7f3653596640 10 librados: watch_flush enter 2024-03-25T19:21:08.651 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.650+0000 7f3653596640 10 librados: watch_flush exit 2024-03-25T19:21:08.651 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.650+0000 7f3653596640 20 client.4288.objecter shutdown clearing up homeless session... 2024-03-25T19:21:08.651 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.650+0000 7f3653596640 10 client.4288.objecter successfully canceled tick 2024-03-25T19:21:08.651 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.651+0000 7f3653596640 1 -- 172.21.15.83:0/2660917450 >> v1:172.21.15.103:6800/345072327 conn(0x7f363006df70 legacy=0x7f3630070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:08.651 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.651+0000 7f3653596640 10 monclient: shutdown 2024-03-25T19:21:08.651 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.651+0000 7f3653596640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:21:08.651 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.651+0000 7f3653596640 1 -- 172.21.15.83:0/2660917450 >> v1:172.21.15.103:6789/0 conn(0x7f364c0a98f0 legacy=0x7f364c1dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:08.651 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.651+0000 7f3653596640 1 -- 172.21.15.83:0/2660917450 shutdown_connections 2024-03-25T19:21:08.651 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.651+0000 7f3653596640 1 -- 172.21.15.83:0/2660917450 >> 172.21.15.83:0/2660917450 conn(0x7f364c0b7ea0 msgr2=0x7f364c14ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:21:08.651 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.651+0000 7f3653596640 1 -- 172.21.15.83:0/2660917450 shutdown_connections 2024-03-25T19:21:08.651 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.651+0000 7f3653596640 1 -- 172.21.15.83:0/2660917450 wait complete. 2024-03-25T19:21:08.652 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:08.651+0000 7f3653596640 1 librados: shutdown 2024-03-25T19:21:08.663 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:21:15.469 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:21:15.585 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.585+0000 7fdcbc793640 10 monclient: get_monmap_and_config 2024-03-25T19:21:15.585 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.585+0000 7fdcbc793640 10 monclient: build_initial_monmap 2024-03-25T19:21:15.585 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.585+0000 7fdcbc793640 10 monclient: monmap: 2024-03-25T19:21:15.585 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:21:15.585 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:21:15.585 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:21:15.586463+0000 2024-03-25T19:21:15.585 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:21:15.586463+0000 2024-03-25T19:21:15.586 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:21:15.586 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:21:15.586 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:21:15.586 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:21:15.586 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:21:15.586 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:15.587 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.586+0000 7fdcbc793640 1 Processor -- start 2024-03-25T19:21:15.587 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.586+0000 7fdcbc793640 1 -- start start 2024-03-25T19:21:15.587 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.586+0000 7fdcbc793640 10 monclient: init 2024-03-25T19:21:15.588 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.587+0000 7fdcbc793640 10 monclient: _reopen_session rank -1 2024-03-25T19:21:15.588 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.587+0000 7fdcbc793640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:21:15.588 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.587+0000 7fdcbc793640 10 monclient(hunting): picked mon.noname-a con 0x7fdcb40a98f0 addr v1:172.21.15.83:6789/0 2024-03-25T19:21:15.588 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.587+0000 7fdcbc793640 10 monclient(hunting): picked mon.noname-c con 0x7fdcb413f710 addr v1:172.21.15.103:6789/0 2024-03-25T19:21:15.588 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.587+0000 7fdcbc793640 10 monclient(hunting): picked mon.noname-b con 0x7fdcb41430c0 addr v1:172.21.15.83:6790/0 2024-03-25T19:21:15.588 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.587+0000 7fdcbc793640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcb4142520 con 0x7fdcb40a98f0 2024-03-25T19:21:15.588 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.587+0000 7fdcbc793640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcb40b28a0 con 0x7fdcb413f710 2024-03-25T19:21:15.588 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.587+0000 7fdcbc793640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcb40b2a30 con 0x7fdcb41430c0 2024-03-25T19:21:15.588 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.587+0000 7fdcbc793640 10 monclient(hunting): _renew_subs 2024-03-25T19:21:15.588 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.587+0000 7fdcbc793640 10 monclient(hunting): authenticate will time out at 6132.859353s 2024-03-25T19:21:15.601 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.602+0000 7fdcba52f640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fdcb40a98f0 0x7fdcb40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:55790/0 (socket says 172.21.15.83:55790) 2024-03-25T19:21:15.602 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.602+0000 7fdcba52f640 1 -- 172.21.15.83:0/3762846397 learned_addr learned my addr 172.21.15.83:0/3762846397 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:21:15.611 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.612+0000 7fdcb952d640 1 -- 172.21.15.83:0/3762846397 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2754910650 0 0) 0x7fdcb4142520 con 0x7fdcb40a98f0 2024-03-25T19:21:15.611 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.612+0000 7fdcb952d640 10 monclient(hunting): _negotiate 2024-03-25T19:21:15.611 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.612+0000 7fdcb952d640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:15.612 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.612+0000 7fdcb952d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:15.612 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.612+0000 7fdcb952d640 10 monclient(hunting): my global_id is 4284 2024-03-25T19:21:15.612 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.612+0000 7fdcb952d640 1 -- 172.21.15.83:0/3762846397 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdca0003520 con 0x7fdcb40a98f0 2024-03-25T19:21:15.618 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.618+0000 7fdcb952d640 1 -- 172.21.15.83:0/3762846397 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4173053013 0 0) 0x7fdcb40b28a0 con 0x7fdcb413f710 2024-03-25T19:21:15.618 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.618+0000 7fdcb952d640 10 monclient(hunting): _negotiate 2024-03-25T19:21:15.618 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.618+0000 7fdcb952d640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:15.618 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.618+0000 7fdcb952d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:15.618 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.618+0000 7fdcb952d640 10 monclient(hunting): my global_id is 4291 2024-03-25T19:21:15.618 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.618+0000 7fdcb952d640 1 -- 172.21.15.83:0/3762846397 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcb4142520 con 0x7fdcb413f710 2024-03-25T19:21:15.625 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.625+0000 7fdcb952d640 1 -- 172.21.15.83:0/3762846397 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3464343422 0 0) 0x7fdcb40b2a30 con 0x7fdcb41430c0 2024-03-25T19:21:15.625 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.625+0000 7fdcb952d640 10 monclient(hunting): _negotiate 2024-03-25T19:21:15.625 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.625+0000 7fdcb952d640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:15.625 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.625+0000 7fdcb952d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:15.625 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.625+0000 7fdcb952d640 10 monclient(hunting): my global_id is 4295 2024-03-25T19:21:15.625 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.625+0000 7fdcb952d640 1 -- 172.21.15.83:0/3762846397 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcb40b28a0 con 0x7fdcb41430c0 2024-03-25T19:21:15.634 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.634+0000 7fdcb952d640 1 -- 172.21.15.83:0/3762846397 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1067650848 0 0) 0x7fdca0003520 con 0x7fdcb40a98f0 2024-03-25T19:21:15.634 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.634+0000 7fdcb952d640 1 -- 172.21.15.83:0/3762846397 >> v1:172.21.15.83:6790/0 conn(0x7fdcb41430c0 legacy=0x7fdcb414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:15.634 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.634+0000 7fdcb952d640 1 -- 172.21.15.83:0/3762846397 >> v1:172.21.15.103:6789/0 conn(0x7fdcb413f710 legacy=0x7fdcb41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:15.634 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.634+0000 7fdcb952d640 10 monclient: _finish_hunting 0 2024-03-25T19:21:15.635 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.634+0000 7fdcb952d640 1 monclient: found mon.noname-a 2024-03-25T19:21:15.635 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.634+0000 7fdcb952d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:21:15.635 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.634+0000 7fdcb952d640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.83:6789/0 2024-03-25T19:21:15.635 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.634+0000 7fdcb952d640 1 -- 172.21.15.83:0/3762846397 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcb40b44c0 con 0x7fdcb40a98f0 2024-03-25T19:21:15.635 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.634+0000 7fdcb952d640 10 monclient: _finish_auth 0 2024-03-25T19:21:15.635 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.634+0000 7fdcb952d640 10 monclient: _check_auth_tickets 2024-03-25T19:21:15.635 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.634+0000 7fdcb952d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:21:15.635 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.634+0000 7fdcb952d640 1 -- 172.21.15.83:0/3762846397 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fdcb0001690 con 0x7fdcb40a98f0 2024-03-25T19:21:15.635 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.634+0000 7fdcbc793640 5 monclient: authenticate success, global_id 4284 2024-03-25T19:21:15.635 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.634+0000 7fdcbc793640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:21:15.635 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.634+0000 7fdcb952d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:15.635 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.634+0000 7fdcb952d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:21:15.635 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.634+0000 7fdcb952d640 10 monclient: dump: 2024-03-25T19:21:15.635 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:21:15.635 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:21:15.635 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:15.636 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:15.636 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:15.636 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:21:15.636 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:21:15.636 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:21:15.636 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:21:15.636 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:15.636 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.634+0000 7fdcbc793640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:21:15.652 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.653+0000 7fdcb952d640 1 -- 172.21.15.83:0/3762846397 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdcb00017f0 con 0x7fdcb40a98f0 2024-03-25T19:21:15.653 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.653+0000 7fdcb952d640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:21:15.653 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.653+0000 7fdcb952d640 1 -- 172.21.15.83:0/3762846397 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fdcb0001c00 con 0x7fdcb40a98f0 2024-03-25T19:21:15.653 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.653+0000 7fdcbc793640 10 monclient: get_monmap_and_config success 2024-03-25T19:21:15.653 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.653+0000 7fdcbc793640 4 set_mon_vals no callback set 2024-03-25T19:21:15.653 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.653+0000 7fdcb952d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:15.653 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.653+0000 7fdcb952d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:21:15.653 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.653+0000 7fdcb952d640 10 monclient: dump: 2024-03-25T19:21:15.653 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:21:15.653 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:21:15.653 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:15.653 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:15.653 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:15.653 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:21:15.653 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:21:15.653 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:21:15.653 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:21:15.654 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:15.655 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.654+0000 7fdcbc793640 10 monclient: shutdown 2024-03-25T19:21:15.655 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.654+0000 7fdcbc793640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:21:15.655 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.654+0000 7fdcbc793640 1 -- 172.21.15.83:0/3762846397 >> v1:172.21.15.83:6789/0 conn(0x7fdcb40a98f0 legacy=0x7fdcb40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:15.655 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.654+0000 7fdcbc793640 1 -- 172.21.15.83:0/3762846397 shutdown_connections 2024-03-25T19:21:15.655 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.654+0000 7fdcbc793640 1 -- 172.21.15.83:0/3762846397 >> 172.21.15.83:0/3762846397 conn(0x7fdcb40b7ea0 msgr2=0x7fdcb40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:21:15.655 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.654+0000 7fdcbc793640 1 -- 172.21.15.83:0/3762846397 shutdown_connections 2024-03-25T19:21:15.655 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.654+0000 7fdcbc793640 1 -- 172.21.15.83:0/3762846397 wait complete. 2024-03-25T19:21:15.655 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.655+0000 7fdcbc793640 10 monclient: build_initial_monmap 2024-03-25T19:21:15.655 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.655+0000 7fdcbc793640 10 monclient: monmap: 2024-03-25T19:21:15.655 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:21:15.655 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:21:15.656 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:21:15.656 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:21:15.656 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:21:15.656 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:21:15.656 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:21:15.656 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:21:15.656 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:21:15.656 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:15.656 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.655+0000 7fdcbc793640 1 librados: starting msgr at 2024-03-25T19:21:15.656 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.655+0000 7fdcbc793640 1 librados: starting objecter 2024-03-25T19:21:15.656 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.655+0000 7fdcbc793640 1 Processor -- start 2024-03-25T19:21:15.657 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.656+0000 7fdcbc793640 1 -- start start 2024-03-25T19:21:15.657 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.656+0000 7fdcbc793640 1 librados: setting wanted keys 2024-03-25T19:21:15.657 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.656+0000 7fdcbc793640 1 librados: calling monclient init 2024-03-25T19:21:15.657 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.656+0000 7fdcbc793640 10 monclient: init 2024-03-25T19:21:15.657 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.656+0000 7fdcbc793640 10 monclient: _reopen_session rank -1 2024-03-25T19:21:15.658 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.656+0000 7fdcbc793640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:21:15.658 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.656+0000 7fdcbc793640 10 monclient(hunting): picked mon.noname-a con 0x7fdcb40a98f0 addr v1:172.21.15.83:6789/0 2024-03-25T19:21:15.658 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.656+0000 7fdcbc793640 10 monclient(hunting): picked mon.noname-b con 0x7fdcb413f710 addr v1:172.21.15.103:6789/0 2024-03-25T19:21:15.658 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.656+0000 7fdcbc793640 10 monclient(hunting): picked mon.noname-c con 0x7fdcb41430c0 addr v1:172.21.15.83:6790/0 2024-03-25T19:21:15.658 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.656+0000 7fdcbc793640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcb40701b0 con 0x7fdcb40a98f0 2024-03-25T19:21:15.658 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.656+0000 7fdcbc793640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcb4070400 con 0x7fdcb413f710 2024-03-25T19:21:15.658 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.656+0000 7fdcbc793640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcb40706e0 con 0x7fdcb41430c0 2024-03-25T19:21:15.658 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.656+0000 7fdcbc793640 10 monclient(hunting): _renew_subs 2024-03-25T19:21:15.658 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.656+0000 7fdcbc793640 10 monclient(hunting): authenticate will time out at 6132.928722s 2024-03-25T19:21:15.667 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.668+0000 7fdcb9d2e640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fdcb413f710 0x7fdcb41381f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:52684/0 (socket says 172.21.15.83:52684) 2024-03-25T19:21:15.668 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.668+0000 7fdcb9d2e640 1 -- 172.21.15.83:0/184511270 learned_addr learned my addr 172.21.15.83:0/184511270 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:21:15.669 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.669+0000 7fdcaaffd640 10 client.?.objecter ms_handle_connect 0x7fdcb413f710 2024-03-25T19:21:15.669 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.669+0000 7fdcaaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:15.674 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.675+0000 7fdcaaffd640 10 client.?.objecter ms_handle_connect 0x7fdcb40a98f0 2024-03-25T19:21:15.674 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.675+0000 7fdcaaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:15.678 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.678+0000 7fdcaaffd640 10 client.?.objecter ms_handle_connect 0x7fdcb41430c0 2024-03-25T19:21:15.678 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.678+0000 7fdcaaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:15.681 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.681+0000 7fdcaaffd640 1 -- 172.21.15.83:0/184511270 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3175128140 0 0) 0x7fdcb40706e0 con 0x7fdcb41430c0 2024-03-25T19:21:15.681 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.681+0000 7fdcaaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:21:15.681 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.681+0000 7fdcaaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:15.681 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.681+0000 7fdcaaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:15.681 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.681+0000 7fdcaaffd640 10 monclient(hunting): my global_id is 4298 2024-03-25T19:21:15.681 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.681+0000 7fdcaaffd640 1 -- 172.21.15.83:0/184511270 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc98004eb0 con 0x7fdcb41430c0 2024-03-25T19:21:15.686 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.687+0000 7fdcaaffd640 1 -- 172.21.15.83:0/184511270 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 934010864 0 0) 0x7fdcb40701b0 con 0x7fdcb40a98f0 2024-03-25T19:21:15.687 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.687+0000 7fdcaaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:21:15.687 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.687+0000 7fdcaaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:15.687 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.687+0000 7fdcaaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:15.687 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.687+0000 7fdcaaffd640 10 monclient(hunting): my global_id is 4287 2024-03-25T19:21:15.687 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.687+0000 7fdcaaffd640 1 -- 172.21.15.83:0/184511270 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcb40706e0 con 0x7fdcb40a98f0 2024-03-25T19:21:15.687 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.688+0000 7fdcaaffd640 1 -- 172.21.15.83:0/184511270 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 288089316 0 0) 0x7fdcb4070400 con 0x7fdcb413f710 2024-03-25T19:21:15.687 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.688+0000 7fdcaaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:21:15.687 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.688+0000 7fdcaaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:15.688 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.688+0000 7fdcaaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:15.688 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.688+0000 7fdcaaffd640 10 monclient(hunting): my global_id is 4294 2024-03-25T19:21:15.688 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.688+0000 7fdcaaffd640 1 -- 172.21.15.83:0/184511270 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcb40701b0 con 0x7fdcb413f710 2024-03-25T19:21:15.694 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.694+0000 7fdcaaffd640 1 -- 172.21.15.83:0/184511270 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3031447119 0 0) 0x7fdc98004eb0 con 0x7fdcb41430c0 2024-03-25T19:21:15.694 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.694+0000 7fdcaaffd640 1 -- 172.21.15.83:0/184511270 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdcb4070400 con 0x7fdcb41430c0 2024-03-25T19:21:15.694 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.695+0000 7fdcaaffd640 1 -- 172.21.15.83:0/184511270 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fdcac001bd0 con 0x7fdcb41430c0 2024-03-25T19:21:15.694 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.695+0000 7fdcaaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:15.695 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.695+0000 7fdcaaffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:21:15.695 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.695+0000 7fdcaaffd640 10 monclient(hunting): dump: 2024-03-25T19:21:15.695 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:21:15.695 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:21:15.695 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:15.695 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:15.695 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:15.695 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:21:15.695 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:21:15.695 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:21:15.695 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:21:15.695 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:15.695 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.695+0000 7fdcaaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:21:15.702 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.702+0000 7fdcaaffd640 1 -- 172.21.15.83:0/184511270 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1866832515 0 0) 0x7fdcb40701b0 con 0x7fdcb413f710 2024-03-25T19:21:15.702 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.702+0000 7fdcaaffd640 1 -- 172.21.15.83:0/184511270 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdc98004eb0 con 0x7fdcb413f710 2024-03-25T19:21:15.702 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.702+0000 7fdcaaffd640 1 -- 172.21.15.83:0/184511270 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fdca4001930 con 0x7fdcb413f710 2024-03-25T19:21:15.702 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.702+0000 7fdcaaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:15.702 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.702+0000 7fdcaaffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:21:15.702 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.702+0000 7fdcaaffd640 10 monclient(hunting): dump: 2024-03-25T19:21:15.702 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:21:15.702 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:21:15.702 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:15.702 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:15.702 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:15.703 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:21:15.703 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:21:15.703 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:21:15.703 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:21:15.703 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:15.714 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.714+0000 7fdcaaffd640 1 -- 172.21.15.83:0/184511270 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1014910212 0 0) 0x7fdcb4070400 con 0x7fdcb41430c0 2024-03-25T19:21:15.714 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.714+0000 7fdcaaffd640 1 -- 172.21.15.83:0/184511270 >> v1:172.21.15.103:6789/0 conn(0x7fdcb413f710 legacy=0x7fdcb41381f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:15.714 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.714+0000 7fdcaaffd640 1 -- 172.21.15.83:0/184511270 >> v1:172.21.15.83:6789/0 conn(0x7fdcb40a98f0 legacy=0x7fdcb4070de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:15.714 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.714+0000 7fdcaaffd640 10 monclient: _finish_hunting 0 2024-03-25T19:21:15.714 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.714+0000 7fdcaaffd640 1 monclient: found mon.c 2024-03-25T19:21:15.714 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.714+0000 7fdcaaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:21:15.714 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.714+0000 7fdcaaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:21:15.714 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.714+0000 7fdcaaffd640 1 -- 172.21.15.83:0/184511270 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcb40709c0 con 0x7fdcb41430c0 2024-03-25T19:21:15.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.714+0000 7fdcaaffd640 10 monclient: _finish_auth 0 2024-03-25T19:21:15.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.714+0000 7fdcaaffd640 10 monclient: _check_auth_tickets 2024-03-25T19:21:15.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.714+0000 7fdcaaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:21:15.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.714+0000 7fdcbc793640 5 monclient: authenticate success, global_id 4298 2024-03-25T19:21:15.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.714+0000 7fdcbc793640 10 monclient: _renew_subs 2024-03-25T19:21:15.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.714+0000 7fdcbc793640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:21:15.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.714+0000 7fdcbc793640 1 -- 172.21.15.83:0/184511270 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcb406f210 con 0x7fdcb41430c0 2024-03-25T19:21:15.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.714+0000 7fdcbc793640 10 client.4298.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:21:15.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.714+0000 7fdcbc793640 10 monclient: _renew_subs 2024-03-25T19:21:15.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.714+0000 7fdcbc793640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:21:15.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.714+0000 7fdcbc793640 1 -- 172.21.15.83:0/184511270 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcb406f730 con 0x7fdcb41430c0 2024-03-25T19:21:15.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.714+0000 7fdcbc793640 1 librados: init done 2024-03-25T19:21:15.716 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.715+0000 7fdcbc793640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:15.716 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.715+0000 7fdcbc793640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:15.716 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.715+0000 7fdcbc793640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:21:15.716 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:15.715+0000 7fdcbc793640 1 -- 172.21.15.83:0/184511270 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcb40627b0 con 0x7fdcb41430c0 2024-03-25T19:21:16.026 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:16.026+0000 7fdcaaffd640 1 -- 172.21.15.83:0/184511270 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdcac002060 con 0x7fdcb41430c0 2024-03-25T19:21:16.026 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:16.026+0000 7fdcaaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:21:16.026 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:16.026+0000 7fdcaaffd640 1 -- 172.21.15.83:0/184511270 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fdcac003b30 con 0x7fdcb41430c0 2024-03-25T19:21:16.026 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:16.026+0000 7fdcaaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:16.026 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:16.026+0000 7fdcaaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:21:16.026 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:16.026+0000 7fdcaaffd640 10 monclient: dump: 2024-03-25T19:21:16.027 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:21:16.027 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:21:16.027 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:16.027 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:16.027 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:16.027 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:21:16.027 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:21:16.027 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:21:16.027 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:21:16.027 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:16.027 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:16.026+0000 7fdcb8d2c640 4 set_mon_vals no callback set 2024-03-25T19:21:16.028 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:16.027+0000 7fdcaaffd640 1 -- 172.21.15.83:0/184511270 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7fdcac003d50 con 0x7fdcb41430c0 2024-03-25T19:21:16.028 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:16.027+0000 7fdcba52f640 10 monclient: get_auth_request con 0x7fdc9806df20 auth_method 0 2024-03-25T19:21:16.028 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:16.028+0000 7fdcaaffd640 10 client.4298.objecter ms_handle_connect 0x7fdc9806df20 2024-03-25T19:21:16.033 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:16.033+0000 7fdcaaffd640 1 -- 172.21.15.83:0/184511270 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7fdcac0538a0 con 0x7fdcb41430c0 2024-03-25T19:21:16.033 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:16.033+0000 7fdcaaffd640 10 client.4298.objecter ms_dispatch 0x7fdcb4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:21:16.033 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:16.033+0000 7fdcaaffd640 3 client.4298.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:21:16.033 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:16.033+0000 7fdcaaffd640 3 client.4298.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:21:16.033 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:16.033+0000 7fdcaaffd640 20 client.4298.objecter dump_active .. 0 homeless 2024-03-25T19:21:16.033 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:16.033+0000 7fdcaaffd640 1 -- 172.21.15.83:0/184511270 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdcac04f3b0 con 0x7fdcb41430c0 2024-03-25T19:21:16.033 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:16.033+0000 7fdcaaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:16.033 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:16.033+0000 7fdcaaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:21:16.268 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:16.268+0000 7fdcbc793640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:16.268 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:16.268+0000 7fdcbc793640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:16.268 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:16.268+0000 7fdcbc793640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:21:16.268 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:16.268+0000 7fdcbc793640 1 -- 172.21.15.83:0/184511270 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdcb4074cc0 con 0x7fdcb41430c0 2024-03-25T19:21:16.326 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:16.326+0000 7fdcaaffd640 1 -- 172.21.15.83:0/184511270 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7fdcac0512e0 con 0x7fdcb41430c0 2024-03-25T19:21:16.326 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:16.326+0000 7fdcaaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:16.326 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:16.326+0000 7fdcaaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:21:16.326 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:21:16.327 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:21:16.329 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:16.328+0000 7fdcbc793640 10 librados: watch_flush enter 2024-03-25T19:21:16.329 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:16.328+0000 7fdcbc793640 10 librados: watch_flush exit 2024-03-25T19:21:16.329 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:16.328+0000 7fdcbc793640 20 client.4298.objecter shutdown clearing up homeless session... 2024-03-25T19:21:16.329 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:16.328+0000 7fdcbc793640 10 client.4298.objecter successfully canceled tick 2024-03-25T19:21:16.329 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:16.329+0000 7fdcbc793640 1 -- 172.21.15.83:0/184511270 >> v1:172.21.15.103:6800/345072327 conn(0x7fdc9806df20 legacy=0x7fdc980703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:16.329 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:16.329+0000 7fdcbc793640 10 monclient: shutdown 2024-03-25T19:21:16.329 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:16.329+0000 7fdcbc793640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:21:16.329 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:16.329+0000 7fdcbc793640 1 -- 172.21.15.83:0/184511270 >> v1:172.21.15.83:6790/0 conn(0x7fdcb41430c0 legacy=0x7fdcb41e77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:16.329 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:16.329+0000 7fdcbc793640 1 -- 172.21.15.83:0/184511270 shutdown_connections 2024-03-25T19:21:16.329 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:16.329+0000 7fdcbc793640 1 -- 172.21.15.83:0/184511270 >> 172.21.15.83:0/184511270 conn(0x7fdcb40b7ea0 msgr2=0x7fdcb4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:21:16.329 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:16.329+0000 7fdcbc793640 1 -- 172.21.15.83:0/184511270 shutdown_connections 2024-03-25T19:21:16.329 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:16.329+0000 7fdcbc793640 1 -- 172.21.15.83:0/184511270 wait complete. 2024-03-25T19:21:16.329 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:16.329+0000 7fdcbc793640 1 librados: shutdown 2024-03-25T19:21:16.340 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:21:22.363 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:02:04.290 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:22.376 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:02:04.304 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:22.391 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:02:04.319 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:22.415 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:02:04.343 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:22.442 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:02:04.352 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:22.450 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:02:04.378 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:22.451 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:02:04.361 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:22.475 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:02:04.403 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:22.483 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:02:04.393 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:23.146 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:21:23.258 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.258+0000 7fb71f688640 10 monclient: get_monmap_and_config 2024-03-25T19:21:23.258 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.258+0000 7fb71f688640 10 monclient: build_initial_monmap 2024-03-25T19:21:23.258 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.258+0000 7fb71f688640 10 monclient: monmap: 2024-03-25T19:21:23.258 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:21:23.258 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:21:23.258 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:21:23.259469+0000 2024-03-25T19:21:23.258 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:21:23.259469+0000 2024-03-25T19:21:23.259 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:21:23.259 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:21:23.259 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:21:23.259 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:21:23.259 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:21:23.259 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:23.260 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.259+0000 7fb71f688640 1 Processor -- start 2024-03-25T19:21:23.260 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.259+0000 7fb71f688640 1 -- start start 2024-03-25T19:21:23.261 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.259+0000 7fb71f688640 10 monclient: init 2024-03-25T19:21:23.261 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.259+0000 7fb71f688640 10 monclient: _reopen_session rank -1 2024-03-25T19:21:23.261 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.259+0000 7fb71f688640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:21:23.261 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.259+0000 7fb71f688640 10 monclient(hunting): picked mon.noname-c con 0x7fb7180a98f0 addr v1:172.21.15.103:6789/0 2024-03-25T19:21:23.261 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.260+0000 7fb71f688640 10 monclient(hunting): picked mon.noname-b con 0x7fb71813f710 addr v1:172.21.15.83:6790/0 2024-03-25T19:21:23.261 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.260+0000 7fb71f688640 10 monclient(hunting): picked mon.noname-a con 0x7fb7181430c0 addr v1:172.21.15.83:6789/0 2024-03-25T19:21:23.261 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.260+0000 7fb71f688640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb718142520 con 0x7fb7181430c0 2024-03-25T19:21:23.261 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.260+0000 7fb71f688640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7180b28a0 con 0x7fb7180a98f0 2024-03-25T19:21:23.261 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.260+0000 7fb71f688640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7180b2a30 con 0x7fb71813f710 2024-03-25T19:21:23.261 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.260+0000 7fb71f688640 10 monclient(hunting): _renew_subs 2024-03-25T19:21:23.261 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.260+0000 7fb71f688640 10 monclient(hunting): authenticate will time out at 6140.531938s 2024-03-25T19:21:23.265 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.266+0000 7fb71cc23640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fb71813f710 0x7fb7181429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:38920/0 (socket says 172.21.15.83:38920) 2024-03-25T19:21:23.265 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.266+0000 7fb71cc23640 1 -- 172.21.15.83:0/3855055807 learned_addr learned my addr 172.21.15.83:0/3855055807 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:21:23.276 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.277+0000 7fb70ffff640 1 -- 172.21.15.83:0/3855055807 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3885396787 0 0) 0x7fb7180b2a30 con 0x7fb71813f710 2024-03-25T19:21:23.276 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.277+0000 7fb70ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:21:23.277 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.277+0000 7fb70ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:23.277 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.277+0000 7fb70ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:23.277 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.277+0000 7fb70ffff640 10 monclient(hunting): my global_id is 4301 2024-03-25T19:21:23.277 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.277+0000 7fb70ffff640 1 -- 172.21.15.83:0/3855055807 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb704003520 con 0x7fb71813f710 2024-03-25T19:21:23.290 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.290+0000 7fb70ffff640 1 -- 172.21.15.83:0/3855055807 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2587202440 0 0) 0x7fb704003520 con 0x7fb71813f710 2024-03-25T19:21:23.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.291+0000 7fb70ffff640 1 -- 172.21.15.83:0/3855055807 >> v1:172.21.15.103:6789/0 conn(0x7fb7180a98f0 legacy=0x7fb7180a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:23.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.291+0000 7fb70ffff640 1 -- 172.21.15.83:0/3855055807 >> v1:172.21.15.83:6789/0 conn(0x7fb7181430c0 legacy=0x7fb71814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:23.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.291+0000 7fb70ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:21:23.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.291+0000 7fb70ffff640 1 monclient: found mon.noname-b 2024-03-25T19:21:23.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.291+0000 7fb70ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:21:23.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.291+0000 7fb70ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.83:6790/0 2024-03-25T19:21:23.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.291+0000 7fb70ffff640 1 -- 172.21.15.83:0/3855055807 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7180b44c0 con 0x7fb71813f710 2024-03-25T19:21:23.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.291+0000 7fb70ffff640 10 monclient: _finish_auth 0 2024-03-25T19:21:23.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.291+0000 7fb70ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:21:23.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.291+0000 7fb70ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:21:23.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.291+0000 7fb70ffff640 1 -- 172.21.15.83:0/3855055807 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fb708001690 con 0x7fb71813f710 2024-03-25T19:21:23.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.291+0000 7fb70ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:23.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.291+0000 7fb70ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:21:23.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.291+0000 7fb70ffff640 10 monclient: dump: 2024-03-25T19:21:23.291 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:21:23.291 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:21:23.292 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:23.292 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:23.292 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:23.292 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:21:23.292 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:21:23.292 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:21:23.292 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:21:23.292 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:23.292 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.291+0000 7fb71f688640 5 monclient: authenticate success, global_id 4301 2024-03-25T19:21:23.292 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.291+0000 7fb71f688640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:21:23.319 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.320+0000 7fb70ffff640 1 -- 172.21.15.83:0/3855055807 <== mon.2 v1:172.21.15.83:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb7080017f0 con 0x7fb71813f710 2024-03-25T19:21:23.320 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.320+0000 7fb70ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:21:23.320 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.320+0000 7fb70ffff640 1 -- 172.21.15.83:0/3855055807 <== mon.2 v1:172.21.15.83:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fb708001c40 con 0x7fb71813f710 2024-03-25T19:21:23.320 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.320+0000 7fb71f688640 10 monclient: get_monmap_and_config success 2024-03-25T19:21:23.320 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.320+0000 7fb71f688640 4 set_mon_vals no callback set 2024-03-25T19:21:23.320 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.320+0000 7fb70ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:23.320 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.320+0000 7fb70ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:21:23.320 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.320+0000 7fb70ffff640 10 monclient: dump: 2024-03-25T19:21:23.320 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:21:23.320 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:21:23.320 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:23.320 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:23.320 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:23.320 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:21:23.320 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:21:23.320 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:21:23.320 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:21:23.321 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:23.323 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.321+0000 7fb71f688640 10 monclient: shutdown 2024-03-25T19:21:23.323 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.321+0000 7fb71f688640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:21:23.323 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.321+0000 7fb71f688640 1 -- 172.21.15.83:0/3855055807 >> v1:172.21.15.83:6790/0 conn(0x7fb71813f710 legacy=0x7fb7181429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:23.323 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.321+0000 7fb71f688640 1 -- 172.21.15.83:0/3855055807 shutdown_connections 2024-03-25T19:21:23.323 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.321+0000 7fb71f688640 1 -- 172.21.15.83:0/3855055807 >> 172.21.15.83:0/3855055807 conn(0x7fb7180b7ea0 msgr2=0x7fb7180b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:21:23.323 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.321+0000 7fb71f688640 1 -- 172.21.15.83:0/3855055807 shutdown_connections 2024-03-25T19:21:23.323 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.321+0000 7fb71f688640 1 -- 172.21.15.83:0/3855055807 wait complete. 2024-03-25T19:21:23.323 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.322+0000 7fb71f688640 10 monclient: build_initial_monmap 2024-03-25T19:21:23.323 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.322+0000 7fb71f688640 10 monclient: monmap: 2024-03-25T19:21:23.323 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:21:23.323 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:21:23.323 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:21:23.323 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:21:23.324 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:21:23.324 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:21:23.324 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:21:23.324 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:21:23.324 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:21:23.324 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:23.324 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.322+0000 7fb71f688640 1 librados: starting msgr at 2024-03-25T19:21:23.324 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.322+0000 7fb71f688640 1 librados: starting objecter 2024-03-25T19:21:23.324 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.322+0000 7fb71f688640 1 Processor -- start 2024-03-25T19:21:23.324 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.322+0000 7fb71f688640 1 -- start start 2024-03-25T19:21:23.324 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.322+0000 7fb71f688640 1 librados: setting wanted keys 2024-03-25T19:21:23.324 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.322+0000 7fb71f688640 1 librados: calling monclient init 2024-03-25T19:21:23.324 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.322+0000 7fb71f688640 10 monclient: init 2024-03-25T19:21:23.324 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.323+0000 7fb71f688640 10 monclient: _reopen_session rank -1 2024-03-25T19:21:23.324 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.323+0000 7fb71f688640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:21:23.324 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.323+0000 7fb71f688640 10 monclient(hunting): picked mon.noname-c con 0x7fb7180a98f0 addr v1:172.21.15.83:6790/0 2024-03-25T19:21:23.324 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.323+0000 7fb71f688640 10 monclient(hunting): picked mon.noname-b con 0x7fb71813f710 addr v1:172.21.15.103:6789/0 2024-03-25T19:21:23.325 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.323+0000 7fb71f688640 10 monclient(hunting): picked mon.noname-a con 0x7fb7181430c0 addr v1:172.21.15.83:6789/0 2024-03-25T19:21:23.325 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.323+0000 7fb71f688640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb718139a40 con 0x7fb7181430c0 2024-03-25T19:21:23.325 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.323+0000 7fb71f688640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb718133f20 con 0x7fb71813f710 2024-03-25T19:21:23.325 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.323+0000 7fb71f688640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7180ab950 con 0x7fb7180a98f0 2024-03-25T19:21:23.325 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.323+0000 7fb71f688640 10 monclient(hunting): _renew_subs 2024-03-25T19:21:23.325 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.323+0000 7fb71f688640 10 monclient(hunting): authenticate will time out at 6140.595188s 2024-03-25T19:21:23.331 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.331+0000 7fb71dc25640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fb7181430c0 0x7fb718133810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60360/0 (socket says 172.21.15.83:60360) 2024-03-25T19:21:23.331 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.331+0000 7fb71dc25640 1 -- 172.21.15.83:0/1801377097 learned_addr learned my addr 172.21.15.83:0/1801377097 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:21:23.338 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.339+0000 7fb70dffb640 10 client.?.objecter ms_handle_connect 0x7fb7180a98f0 2024-03-25T19:21:23.338 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.339+0000 7fb70dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:23.339 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.340+0000 7fb70dffb640 10 client.?.objecter ms_handle_connect 0x7fb71813f710 2024-03-25T19:21:23.340 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.340+0000 7fb70dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:23.344 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.345+0000 7fb70dffb640 10 client.?.objecter ms_handle_connect 0x7fb7181430c0 2024-03-25T19:21:23.344 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.345+0000 7fb70dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:23.351 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.351+0000 7fb70dffb640 1 -- 172.21.15.83:0/1801377097 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1524429763 0 0) 0x7fb7180ab950 con 0x7fb7180a98f0 2024-03-25T19:21:23.351 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.352+0000 7fb70dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:21:23.351 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.352+0000 7fb70dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:23.351 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.352+0000 7fb70dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:23.351 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.352+0000 7fb70dffb640 10 monclient(hunting): my global_id is 4304 2024-03-25T19:21:23.352 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.352+0000 7fb70dffb640 1 -- 172.21.15.83:0/1801377097 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6fc004eb0 con 0x7fb7180a98f0 2024-03-25T19:21:23.362 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.363+0000 7fb70dffb640 1 -- 172.21.15.83:0/1801377097 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1041872320 0 0) 0x7fb718139a40 con 0x7fb7181430c0 2024-03-25T19:21:23.362 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.363+0000 7fb70dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:21:23.362 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.363+0000 7fb70dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:23.363 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.363+0000 7fb70dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:23.363 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.363+0000 7fb70dffb640 10 monclient(hunting): my global_id is 4293 2024-03-25T19:21:23.363 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.363+0000 7fb70dffb640 1 -- 172.21.15.83:0/1801377097 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7180ab950 con 0x7fb7181430c0 2024-03-25T19:21:23.380 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.380+0000 7fb70dffb640 1 -- 172.21.15.83:0/1801377097 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3756700951 0 0) 0x7fb7180ab950 con 0x7fb7181430c0 2024-03-25T19:21:23.380 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.380+0000 7fb70dffb640 1 -- 172.21.15.83:0/1801377097 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb718139a40 con 0x7fb7181430c0 2024-03-25T19:21:23.380 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.380+0000 7fb70dffb640 1 -- 172.21.15.83:0/1801377097 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fb710001c10 con 0x7fb7181430c0 2024-03-25T19:21:23.380 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.380+0000 7fb70dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:23.380 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.380+0000 7fb70dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:21:23.380 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.380+0000 7fb70dffb640 10 monclient(hunting): dump: 2024-03-25T19:21:23.380 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:21:23.380 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:21:23.381 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:23.381 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:23.381 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:23.381 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:21:23.381 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:21:23.381 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:21:23.381 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:21:23.381 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:23.381 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.380+0000 7fb70dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:21:23.446 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.446+0000 7fb70dffb640 1 -- 172.21.15.83:0/1801377097 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1001959247 0 0) 0x7fb6fc004eb0 con 0x7fb7180a98f0 2024-03-25T19:21:23.446 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.446+0000 7fb70dffb640 1 -- 172.21.15.83:0/1801377097 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7180ab950 con 0x7fb7180a98f0 2024-03-25T19:21:23.446 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.446+0000 7fb70dffb640 1 -- 172.21.15.83:0/1801377097 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fb714001930 con 0x7fb7180a98f0 2024-03-25T19:21:23.446 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.446+0000 7fb70dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:23.446 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.446+0000 7fb70dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:21:23.446 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.446+0000 7fb70dffb640 10 monclient(hunting): dump: 2024-03-25T19:21:23.446 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:21:23.446 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:21:23.446 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:23.446 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:23.446 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:23.447 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:21:23.447 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:21:23.447 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:21:23.447 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:21:23.447 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:23.452 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.451+0000 7fb70dffb640 1 -- 172.21.15.83:0/1801377097 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 45038504 0 0) 0x7fb718139a40 con 0x7fb7181430c0 2024-03-25T19:21:23.452 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.451+0000 7fb70dffb640 1 -- 172.21.15.83:0/1801377097 >> v1:172.21.15.83:6790/0 conn(0x7fb7180a98f0 legacy=0x7fb718138c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:23.452 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.452+0000 7fb70dffb640 1 -- 172.21.15.83:0/1801377097 >> v1:172.21.15.103:6789/0 conn(0x7fb71813f710 legacy=0x7fb718133100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:23.452 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.452+0000 7fb70dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:21:23.452 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.452+0000 7fb70dffb640 1 monclient: found mon.a 2024-03-25T19:21:23.452 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.452+0000 7fb70dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:21:23.452 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.452+0000 7fb70dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:21:23.452 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.452+0000 7fb70dffb640 1 -- 172.21.15.83:0/1801377097 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7180abc30 con 0x7fb7181430c0 2024-03-25T19:21:23.452 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.452+0000 7fb70dffb640 10 monclient: _finish_auth 0 2024-03-25T19:21:23.452 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.452+0000 7fb70dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:21:23.452 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.452+0000 7fb70dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:21:23.452 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.452+0000 7fb71f688640 5 monclient: authenticate success, global_id 4293 2024-03-25T19:21:23.453 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.452+0000 7fb71f688640 10 monclient: _renew_subs 2024-03-25T19:21:23.453 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.452+0000 7fb71f688640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:21:23.453 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.452+0000 7fb71f688640 1 -- 172.21.15.83:0/1801377097 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7180abf10 con 0x7fb7181430c0 2024-03-25T19:21:23.453 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.452+0000 7fb71f688640 10 client.4293.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:21:23.453 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.452+0000 7fb71f688640 10 monclient: _renew_subs 2024-03-25T19:21:23.453 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.452+0000 7fb71f688640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:21:23.453 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.452+0000 7fb71f688640 1 -- 172.21.15.83:0/1801377097 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb718139370 con 0x7fb7181430c0 2024-03-25T19:21:23.453 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.452+0000 7fb71f688640 1 librados: init done 2024-03-25T19:21:23.454 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.453+0000 7fb71f688640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:23.454 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.453+0000 7fb71f688640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:23.454 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.453+0000 7fb71f688640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:21:23.454 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.453+0000 7fb71f688640 1 -- 172.21.15.83:0/1801377097 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb718138980 con 0x7fb7181430c0 2024-03-25T19:21:23.707 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.708+0000 7fb70dffb640 1 -- 172.21.15.83:0/1801377097 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb710003990 con 0x7fb7181430c0 2024-03-25T19:21:23.708 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.708+0000 7fb70dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:21:23.708 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.708+0000 7fb70dffb640 1 -- 172.21.15.83:0/1801377097 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fb710002100 con 0x7fb7181430c0 2024-03-25T19:21:23.708 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.708+0000 7fb70dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:23.708 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.708+0000 7fb70dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:21:23.708 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.708+0000 7fb70dffb640 10 monclient: dump: 2024-03-25T19:21:23.708 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:21:23.708 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:21:23.708 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:23.708 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:23.708 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:23.708 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:21:23.708 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:21:23.709 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:21:23.709 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:21:23.709 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:23.709 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.708+0000 7fb70f7fe640 4 set_mon_vals no callback set 2024-03-25T19:21:23.709 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.709+0000 7fb70dffb640 1 -- 172.21.15.83:0/1801377097 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7fb710003e20 con 0x7fb7181430c0 2024-03-25T19:21:23.709 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.710+0000 7fb71d424640 10 monclient: get_auth_request con 0x7fb6fc06de90 auth_method 0 2024-03-25T19:21:23.710 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.710+0000 7fb70dffb640 10 client.4293.objecter ms_handle_connect 0x7fb6fc06de90 2024-03-25T19:21:23.714 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.714+0000 7fb70dffb640 1 -- 172.21.15.83:0/1801377097 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7fb710054a80 con 0x7fb7181430c0 2024-03-25T19:21:23.714 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.714+0000 7fb70dffb640 10 client.4293.objecter ms_dispatch 0x7fb718000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:21:23.714 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.714+0000 7fb70dffb640 3 client.4293.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:21:23.714 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.714+0000 7fb70dffb640 3 client.4293.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:21:23.714 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.714+0000 7fb70dffb640 20 client.4293.objecter dump_active .. 0 homeless 2024-03-25T19:21:23.714 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.714+0000 7fb70dffb640 1 -- 172.21.15.83:0/1801377097 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb710050590 con 0x7fb7181430c0 2024-03-25T19:21:23.714 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.714+0000 7fb70dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:23.714 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.714+0000 7fb70dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:21:23.953 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.953+0000 7fb71f688640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:23.953 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.953+0000 7fb71f688640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:23.953 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.953+0000 7fb71f688640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:21:23.953 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.953+0000 7fb71f688640 1 -- 172.21.15.83:0/1801377097 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb7180627b0 con 0x7fb7181430c0 2024-03-25T19:21:23.978 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.978+0000 7fb70dffb640 1 -- 172.21.15.83:0/1801377097 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7fb7100524c0 con 0x7fb7181430c0 2024-03-25T19:21:23.978 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.978+0000 7fb70dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:23.978 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.978+0000 7fb70dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:21:23.978 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:21:23.978 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:21:23.980 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.980+0000 7fb71f688640 10 librados: watch_flush enter 2024-03-25T19:21:23.981 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.980+0000 7fb71f688640 10 librados: watch_flush exit 2024-03-25T19:21:23.981 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.980+0000 7fb71f688640 20 client.4293.objecter shutdown clearing up homeless session... 2024-03-25T19:21:23.981 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.980+0000 7fb71f688640 10 client.4293.objecter successfully canceled tick 2024-03-25T19:21:23.981 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.981+0000 7fb71f688640 1 -- 172.21.15.83:0/1801377097 >> v1:172.21.15.103:6800/345072327 conn(0x7fb6fc06de90 legacy=0x7fb6fc070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:23.981 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.981+0000 7fb71f688640 10 monclient: shutdown 2024-03-25T19:21:23.981 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.981+0000 7fb71f688640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:21:23.981 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.981+0000 7fb71f688640 1 -- 172.21.15.83:0/1801377097 >> v1:172.21.15.83:6789/0 conn(0x7fb7181430c0 legacy=0x7fb718133810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:23.981 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.981+0000 7fb71f688640 1 -- 172.21.15.83:0/1801377097 shutdown_connections 2024-03-25T19:21:23.981 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.981+0000 7fb71f688640 1 -- 172.21.15.83:0/1801377097 >> 172.21.15.83:0/1801377097 conn(0x7fb7180b7ea0 msgr2=0x7fb718074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:21:23.981 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.981+0000 7fb71f688640 1 -- 172.21.15.83:0/1801377097 shutdown_connections 2024-03-25T19:21:23.981 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.981+0000 7fb71f688640 1 -- 172.21.15.83:0/1801377097 wait complete. 2024-03-25T19:21:23.981 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:23.981+0000 7fb71f688640 1 librados: shutdown 2024-03-25T19:21:23.992 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:21:30.798 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:21:30.913 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.912+0000 7f55c935d640 10 monclient: get_monmap_and_config 2024-03-25T19:21:30.913 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.912+0000 7f55c935d640 10 monclient: build_initial_monmap 2024-03-25T19:21:30.913 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.912+0000 7f55c935d640 10 monclient: monmap: 2024-03-25T19:21:30.913 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:21:30.914 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:21:30.914 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:21:30.914864+0000 2024-03-25T19:21:30.914 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:21:30.914864+0000 2024-03-25T19:21:30.914 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:21:30.914 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:21:30.914 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:21:30.914 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:21:30.914 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:21:30.914 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:30.915 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.913+0000 7f55c935d640 1 Processor -- start 2024-03-25T19:21:30.915 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.913+0000 7f55c935d640 1 -- start start 2024-03-25T19:21:30.916 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.913+0000 7f55c935d640 10 monclient: init 2024-03-25T19:21:30.916 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.914+0000 7f55c935d640 10 monclient: _reopen_session rank -1 2024-03-25T19:21:30.916 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.914+0000 7f55c935d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:21:30.916 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.914+0000 7f55c935d640 10 monclient(hunting): picked mon.noname-b con 0x7f55c40a98f0 addr v1:172.21.15.83:6790/0 2024-03-25T19:21:30.916 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.914+0000 7f55c935d640 10 monclient(hunting): picked mon.noname-c con 0x7f55c413f710 addr v1:172.21.15.103:6789/0 2024-03-25T19:21:30.916 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.914+0000 7f55c935d640 10 monclient(hunting): picked mon.noname-a con 0x7f55c41430c0 addr v1:172.21.15.83:6789/0 2024-03-25T19:21:30.916 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.914+0000 7f55c935d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c4142520 con 0x7f55c41430c0 2024-03-25T19:21:30.916 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.914+0000 7f55c935d640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c40b28a0 con 0x7f55c413f710 2024-03-25T19:21:30.916 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.914+0000 7f55c935d640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c40b2a30 con 0x7f55c40a98f0 2024-03-25T19:21:30.916 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.914+0000 7f55c935d640 10 monclient(hunting): _renew_subs 2024-03-25T19:21:30.916 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.914+0000 7f55c935d640 10 monclient(hunting): authenticate will time out at 6148.187458s 2024-03-25T19:21:30.925 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.924+0000 7f55c2ffd640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f55c40a98f0 0x7f55c40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:37578/0 (socket says 172.21.15.83:37578) 2024-03-25T19:21:30.925 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.924+0000 7f55c2ffd640 1 -- 172.21.15.83:0/2514540920 learned_addr learned my addr 172.21.15.83:0/2514540920 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:21:30.934 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.933+0000 7f55c1ffb640 1 -- 172.21.15.83:0/2514540920 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1745403231 0 0) 0x7f55c40b28a0 con 0x7f55c413f710 2024-03-25T19:21:30.934 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.933+0000 7f55c1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:21:30.934 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.933+0000 7f55c1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:30.934 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.933+0000 7f55c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:30.934 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.933+0000 7f55c1ffb640 10 monclient(hunting): my global_id is 4300 2024-03-25T19:21:30.934 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.933+0000 7f55c1ffb640 1 -- 172.21.15.83:0/2514540920 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55ac003520 con 0x7f55c413f710 2024-03-25T19:21:30.942 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.941+0000 7f55c1ffb640 1 -- 172.21.15.83:0/2514540920 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2705245434 0 0) 0x7f55c4142520 con 0x7f55c41430c0 2024-03-25T19:21:30.942 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.941+0000 7f55c1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:21:30.942 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.941+0000 7f55c1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:30.942 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.941+0000 7f55c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:30.942 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.941+0000 7f55c1ffb640 10 monclient(hunting): my global_id is 4296 2024-03-25T19:21:30.942 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.941+0000 7f55c1ffb640 1 -- 172.21.15.83:0/2514540920 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55c40b28a0 con 0x7f55c41430c0 2024-03-25T19:21:30.943 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.942+0000 7f55c1ffb640 1 -- 172.21.15.83:0/2514540920 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1433155048 0 0) 0x7f55ac003520 con 0x7f55c413f710 2024-03-25T19:21:30.943 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.942+0000 7f55c1ffb640 1 -- 172.21.15.83:0/2514540920 >> v1:172.21.15.83:6790/0 conn(0x7f55c40a98f0 legacy=0x7f55c40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:30.943 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.942+0000 7f55c1ffb640 1 -- 172.21.15.83:0/2514540920 >> v1:172.21.15.83:6789/0 conn(0x7f55c41430c0 legacy=0x7f55c414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:30.943 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.942+0000 7f55c1ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:21:30.943 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.942+0000 7f55c1ffb640 1 monclient: found mon.noname-c 2024-03-25T19:21:30.943 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.942+0000 7f55c1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:21:30.943 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.942+0000 7f55c1ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.103:6789/0 2024-03-25T19:21:30.943 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.942+0000 7f55c1ffb640 1 -- 172.21.15.83:0/2514540920 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55c40b44c0 con 0x7f55c413f710 2024-03-25T19:21:30.944 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.942+0000 7f55c1ffb640 10 monclient: _finish_auth 0 2024-03-25T19:21:30.944 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.942+0000 7f55c1ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:21:30.944 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.942+0000 7f55c1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:21:30.944 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.942+0000 7f55c1ffb640 1 -- 172.21.15.83:0/2514540920 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f55b8001690 con 0x7f55c413f710 2024-03-25T19:21:30.944 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.942+0000 7f55c1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:30.944 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.942+0000 7f55c1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:21:30.944 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.942+0000 7f55c1ffb640 10 monclient: dump: 2024-03-25T19:21:30.944 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:21:30.944 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:21:30.944 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:30.944 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:30.944 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:30.944 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:21:30.944 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:21:30.944 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:21:30.944 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:21:30.945 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:30.945 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.942+0000 7f55c935d640 5 monclient: authenticate success, global_id 4300 2024-03-25T19:21:30.945 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.942+0000 7f55c935d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:21:30.958 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.958+0000 7f55c1ffb640 1 -- 172.21.15.83:0/2514540920 <== mon.1 v1:172.21.15.103:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f55b8001870 con 0x7f55c413f710 2024-03-25T19:21:30.958 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.958+0000 7f55c1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:21:30.958 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.958+0000 7f55c1ffb640 1 -- 172.21.15.83:0/2514540920 <== mon.1 v1:172.21.15.103:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f55b8001c80 con 0x7f55c413f710 2024-03-25T19:21:30.959 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.958+0000 7f55c935d640 10 monclient: get_monmap_and_config success 2024-03-25T19:21:30.959 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.958+0000 7f55c935d640 4 set_mon_vals no callback set 2024-03-25T19:21:30.959 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.958+0000 7f55c1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:30.959 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.958+0000 7f55c1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:21:30.959 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.958+0000 7f55c1ffb640 10 monclient: dump: 2024-03-25T19:21:30.959 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:21:30.959 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:21:30.959 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:30.959 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:30.959 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:30.959 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:21:30.959 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:21:30.959 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:21:30.959 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:21:30.960 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:30.962 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.958+0000 7f55c935d640 10 monclient: shutdown 2024-03-25T19:21:30.962 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.958+0000 7f55c935d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:21:30.962 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.958+0000 7f55c935d640 1 -- 172.21.15.83:0/2514540920 >> v1:172.21.15.103:6789/0 conn(0x7f55c413f710 legacy=0x7f55c41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:30.962 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.959+0000 7f55c935d640 1 -- 172.21.15.83:0/2514540920 shutdown_connections 2024-03-25T19:21:30.962 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.959+0000 7f55c935d640 1 -- 172.21.15.83:0/2514540920 >> 172.21.15.83:0/2514540920 conn(0x7f55c40b7ea0 msgr2=0x7f55c40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:21:30.962 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.959+0000 7f55c935d640 1 -- 172.21.15.83:0/2514540920 shutdown_connections 2024-03-25T19:21:30.962 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.959+0000 7f55c935d640 1 -- 172.21.15.83:0/2514540920 wait complete. 2024-03-25T19:21:30.962 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.959+0000 7f55c935d640 10 monclient: build_initial_monmap 2024-03-25T19:21:30.962 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.959+0000 7f55c935d640 10 monclient: monmap: 2024-03-25T19:21:30.962 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:21:30.963 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:21:30.963 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:21:30.963 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:21:30.963 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:21:30.963 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:21:30.963 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:21:30.963 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:21:30.963 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:21:30.963 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:30.963 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.960+0000 7f55c935d640 1 librados: starting msgr at 2024-03-25T19:21:30.963 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.960+0000 7f55c935d640 1 librados: starting objecter 2024-03-25T19:21:30.963 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.960+0000 7f55c935d640 1 Processor -- start 2024-03-25T19:21:30.963 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.960+0000 7f55c935d640 1 -- start start 2024-03-25T19:21:30.963 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.960+0000 7f55c935d640 1 librados: setting wanted keys 2024-03-25T19:21:30.963 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.960+0000 7f55c935d640 1 librados: calling monclient init 2024-03-25T19:21:30.964 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.960+0000 7f55c935d640 10 monclient: init 2024-03-25T19:21:30.964 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.961+0000 7f55c935d640 10 monclient: _reopen_session rank -1 2024-03-25T19:21:30.964 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.961+0000 7f55c935d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:21:30.964 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.961+0000 7f55c935d640 10 monclient(hunting): picked mon.noname-b con 0x7f55c40a98f0 addr v1:172.21.15.103:6789/0 2024-03-25T19:21:30.964 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.961+0000 7f55c935d640 10 monclient(hunting): picked mon.noname-a con 0x7f55c413f710 addr v1:172.21.15.83:6789/0 2024-03-25T19:21:30.964 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.961+0000 7f55c935d640 10 monclient(hunting): picked mon.noname-c con 0x7f55c41430c0 addr v1:172.21.15.83:6790/0 2024-03-25T19:21:30.964 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.961+0000 7f55c935d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c406f2c0 con 0x7f55c413f710 2024-03-25T19:21:30.964 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.961+0000 7f55c935d640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c41339d0 con 0x7f55c40a98f0 2024-03-25T19:21:30.964 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.961+0000 7f55c935d640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c406f530 con 0x7f55c41430c0 2024-03-25T19:21:30.964 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.961+0000 7f55c935d640 10 monclient(hunting): _renew_subs 2024-03-25T19:21:30.964 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.961+0000 7f55c935d640 10 monclient(hunting): authenticate will time out at 6148.234172s 2024-03-25T19:21:30.973 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.972+0000 7f55c2ffd640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f55c40a98f0 0x7f55c406f950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:36344/0 (socket says 172.21.15.83:36344) 2024-03-25T19:21:30.973 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.972+0000 7f55c2ffd640 1 -- 172.21.15.83:0/1540898388 learned_addr learned my addr 172.21.15.83:0/1540898388 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:21:30.975 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.975+0000 7f55ab7fe640 10 client.?.objecter ms_handle_connect 0x7f55c40a98f0 2024-03-25T19:21:30.975 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.975+0000 7f55ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:30.979 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.979+0000 7f55ab7fe640 10 client.?.objecter ms_handle_connect 0x7f55c41430c0 2024-03-25T19:21:30.979 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.979+0000 7f55ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:30.980 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.980+0000 7f55ab7fe640 10 client.?.objecter ms_handle_connect 0x7f55c413f710 2024-03-25T19:21:30.980 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.980+0000 7f55ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:30.988 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.987+0000 7f55ab7fe640 1 -- 172.21.15.83:0/1540898388 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3709510596 0 0) 0x7f55c406f530 con 0x7f55c41430c0 2024-03-25T19:21:30.988 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.987+0000 7f55ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:21:30.988 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.987+0000 7f55ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:30.988 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.987+0000 7f55ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:30.988 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.987+0000 7f55ab7fe640 10 monclient(hunting): my global_id is 4310 2024-03-25T19:21:30.988 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.987+0000 7f55ab7fe640 1 -- 172.21.15.83:0/1540898388 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55b0004eb0 con 0x7f55c41430c0 2024-03-25T19:21:30.994 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.993+0000 7f55ab7fe640 1 -- 172.21.15.83:0/1540898388 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3744877698 0 0) 0x7f55c41339d0 con 0x7f55c40a98f0 2024-03-25T19:21:30.994 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.993+0000 7f55ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:21:30.994 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.993+0000 7f55ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:30.994 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.993+0000 7f55ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:30.994 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.993+0000 7f55ab7fe640 10 monclient(hunting): my global_id is 4303 2024-03-25T19:21:30.994 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.993+0000 7f55ab7fe640 1 -- 172.21.15.83:0/1540898388 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55c406f530 con 0x7f55c40a98f0 2024-03-25T19:21:31.000 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.999+0000 7f55ab7fe640 1 -- 172.21.15.83:0/1540898388 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 77180297 0 0) 0x7f55b0004eb0 con 0x7f55c41430c0 2024-03-25T19:21:31.000 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.999+0000 7f55ab7fe640 1 -- 172.21.15.83:0/1540898388 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f55c41339d0 con 0x7f55c41430c0 2024-03-25T19:21:31.000 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.999+0000 7f55ab7fe640 1 -- 172.21.15.83:0/1540898388 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f55bc001bd0 con 0x7f55c41430c0 2024-03-25T19:21:31.000 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.999+0000 7f55ab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:31.000 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.999+0000 7f55ab7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:21:31.000 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.999+0000 7f55ab7fe640 10 monclient(hunting): dump: 2024-03-25T19:21:31.001 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:21:31.001 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:21:31.001 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:31.001 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:31.001 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:31.001 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:21:31.001 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:21:31.001 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:21:31.001 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:21:31.001 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:31.001 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:30.999+0000 7f55ab7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:21:31.002 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.000+0000 7f55ab7fe640 1 -- 172.21.15.83:0/1540898388 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 419611524 0 0) 0x7f55c406f2c0 con 0x7f55c413f710 2024-03-25T19:21:31.002 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.000+0000 7f55ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:21:31.002 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.000+0000 7f55ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:31.002 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.000+0000 7f55ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:31.002 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.000+0000 7f55ab7fe640 10 monclient(hunting): my global_id is 4299 2024-03-25T19:21:31.002 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.000+0000 7f55ab7fe640 1 -- 172.21.15.83:0/1540898388 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55b0004eb0 con 0x7f55c413f710 2024-03-25T19:21:31.011 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.010+0000 7f55ab7fe640 1 -- 172.21.15.83:0/1540898388 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4082813222 0 0) 0x7f55c41339d0 con 0x7f55c41430c0 2024-03-25T19:21:31.011 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.010+0000 7f55ab7fe640 1 -- 172.21.15.83:0/1540898388 >> v1:172.21.15.103:6789/0 conn(0x7f55c40a98f0 legacy=0x7f55c406f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:31.011 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.010+0000 7f55ab7fe640 1 -- 172.21.15.83:0/1540898388 >> v1:172.21.15.83:6789/0 conn(0x7f55c413f710 legacy=0x7f55c4132bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:31.011 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.010+0000 7f55ab7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:21:31.011 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.010+0000 7f55ab7fe640 1 monclient: found mon.c 2024-03-25T19:21:31.011 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.010+0000 7f55ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:21:31.011 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.010+0000 7f55ab7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:21:31.012 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.010+0000 7f55ab7fe640 1 -- 172.21.15.83:0/1540898388 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55c4070060 con 0x7f55c41430c0 2024-03-25T19:21:31.012 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.010+0000 7f55ab7fe640 10 monclient: _finish_auth 0 2024-03-25T19:21:31.012 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.010+0000 7f55ab7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:21:31.012 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.010+0000 7f55ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:21:31.012 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.010+0000 7f55c935d640 5 monclient: authenticate success, global_id 4310 2024-03-25T19:21:31.012 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.010+0000 7f55c935d640 10 monclient: _renew_subs 2024-03-25T19:21:31.012 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.010+0000 7f55c935d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:21:31.012 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.010+0000 7f55c935d640 1 -- 172.21.15.83:0/1540898388 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55c406e380 con 0x7f55c41430c0 2024-03-25T19:21:31.012 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.010+0000 7f55c935d640 10 client.4310.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:21:31.012 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.010+0000 7f55c935d640 10 monclient: _renew_subs 2024-03-25T19:21:31.012 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.010+0000 7f55c935d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:21:31.012 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.010+0000 7f55c935d640 1 -- 172.21.15.83:0/1540898388 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f55c413bc50 con 0x7f55c41430c0 2024-03-25T19:21:31.012 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.010+0000 7f55c935d640 1 librados: init done 2024-03-25T19:21:31.013 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.011+0000 7f55c935d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:31.013 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.011+0000 7f55c935d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:31.013 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.011+0000 7f55c935d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:21:31.013 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.011+0000 7f55c935d640 1 -- 172.21.15.83:0/1540898388 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55c40701f0 con 0x7f55c41430c0 2024-03-25T19:21:31.317 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.316+0000 7f55ab7fe640 1 -- 172.21.15.83:0/1540898388 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f55bc0020c0 con 0x7f55c41430c0 2024-03-25T19:21:31.317 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.316+0000 7f55ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:21:31.318 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.316+0000 7f55ab7fe640 1 -- 172.21.15.83:0/1540898388 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f55bc003b30 con 0x7f55c41430c0 2024-03-25T19:21:31.318 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.316+0000 7f55ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:31.318 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.316+0000 7f55ab7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:21:31.318 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.316+0000 7f55ab7fe640 10 monclient: dump: 2024-03-25T19:21:31.318 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:21:31.318 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:21:31.318 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:31.318 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:31.318 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:31.318 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:21:31.318 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:21:31.318 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:21:31.318 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:21:31.319 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:31.319 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.316+0000 7f55c17fa640 4 set_mon_vals no callback set 2024-03-25T19:21:31.319 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.317+0000 7f55ab7fe640 1 -- 172.21.15.83:0/1540898388 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7f55bc019bc0 con 0x7f55c41430c0 2024-03-25T19:21:31.319 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.318+0000 7f55c2ffd640 10 monclient: get_auth_request con 0x7f55b006de00 auth_method 0 2024-03-25T19:21:31.319 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.318+0000 7f55ab7fe640 10 client.4310.objecter ms_handle_connect 0x7f55b006de00 2024-03-25T19:21:31.324 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.323+0000 7f55ab7fe640 1 -- 172.21.15.83:0/1540898388 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7f55bc053840 con 0x7f55c41430c0 2024-03-25T19:21:31.324 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.323+0000 7f55ab7fe640 10 client.4310.objecter ms_dispatch 0x7f55c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:21:31.324 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.323+0000 7f55ab7fe640 3 client.4310.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:21:31.324 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.323+0000 7f55ab7fe640 3 client.4310.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:21:31.325 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.323+0000 7f55ab7fe640 20 client.4310.objecter dump_active .. 0 homeless 2024-03-25T19:21:31.325 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.323+0000 7f55ab7fe640 1 -- 172.21.15.83:0/1540898388 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f55bc04f180 con 0x7f55c41430c0 2024-03-25T19:21:31.325 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.323+0000 7f55ab7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:31.325 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.323+0000 7f55ab7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:21:31.563 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.562+0000 7f55c935d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:31.563 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.562+0000 7f55c935d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:31.563 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.562+0000 7f55c935d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:21:31.563 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.562+0000 7f55c935d640 1 -- 172.21.15.83:0/1540898388 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f55c413b790 con 0x7f55c41430c0 2024-03-25T19:21:31.596 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.596+0000 7f55ab7fe640 1 -- 172.21.15.83:0/1540898388 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7f55c413b790 con 0x7f55c41430c0 2024-03-25T19:21:31.596 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.596+0000 7f55ab7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:31.596 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.596+0000 7f55ab7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:21:31.597 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:21:31.597 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:21:31.599 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.598+0000 7f55c935d640 10 librados: watch_flush enter 2024-03-25T19:21:31.599 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.598+0000 7f55c935d640 10 librados: watch_flush exit 2024-03-25T19:21:31.599 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.598+0000 7f55c935d640 20 client.4310.objecter shutdown clearing up homeless session... 2024-03-25T19:21:31.599 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.598+0000 7f55c935d640 10 client.4310.objecter successfully canceled tick 2024-03-25T19:21:31.600 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.598+0000 7f55c935d640 1 -- 172.21.15.83:0/1540898388 >> v1:172.21.15.103:6800/345072327 conn(0x7f55b006de00 legacy=0x7f55b00702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:31.600 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.598+0000 7f55c935d640 10 monclient: shutdown 2024-03-25T19:21:31.600 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.598+0000 7f55c935d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:21:31.600 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.598+0000 7f55c935d640 1 -- 172.21.15.83:0/1540898388 >> v1:172.21.15.83:6790/0 conn(0x7f55c41430c0 legacy=0x7f55c41332c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:31.600 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.598+0000 7f55c935d640 1 -- 172.21.15.83:0/1540898388 shutdown_connections 2024-03-25T19:21:31.600 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.598+0000 7f55c935d640 1 -- 172.21.15.83:0/1540898388 >> 172.21.15.83:0/1540898388 conn(0x7f55c40b7ea0 msgr2=0x7f55c4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:21:31.600 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.598+0000 7f55c935d640 1 -- 172.21.15.83:0/1540898388 shutdown_connections 2024-03-25T19:21:31.600 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.598+0000 7f55c935d640 1 -- 172.21.15.83:0/1540898388 wait complete. 2024-03-25T19:21:31.600 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:31.598+0000 7f55c935d640 1 librados: shutdown 2024-03-25T19:21:31.610 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:21:38.417 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:21:38.535 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.535+0000 7f7559e93640 10 monclient: get_monmap_and_config 2024-03-25T19:21:38.535 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.535+0000 7f7559e93640 10 monclient: build_initial_monmap 2024-03-25T19:21:38.535 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.535+0000 7f7559e93640 10 monclient: monmap: 2024-03-25T19:21:38.535 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:21:38.535 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:21:38.535 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:21:38.536390+0000 2024-03-25T19:21:38.535 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:21:38.536390+0000 2024-03-25T19:21:38.535 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:21:38.535 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:21:38.535 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:21:38.536 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:21:38.536 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:21:38.536 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:38.537 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.536+0000 7f7559e93640 1 Processor -- start 2024-03-25T19:21:38.537 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.536+0000 7f7559e93640 1 -- start start 2024-03-25T19:21:38.537 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.536+0000 7f7559e93640 10 monclient: init 2024-03-25T19:21:38.537 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.537+0000 7f7559e93640 10 monclient: _reopen_session rank -1 2024-03-25T19:21:38.537 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.537+0000 7f7559e93640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:21:38.538 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.537+0000 7f7559e93640 10 monclient(hunting): picked mon.noname-c con 0x7f75540a98f0 addr v1:172.21.15.103:6789/0 2024-03-25T19:21:38.538 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.537+0000 7f7559e93640 10 monclient(hunting): picked mon.noname-b con 0x7f755413f710 addr v1:172.21.15.83:6790/0 2024-03-25T19:21:38.538 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.537+0000 7f7559e93640 10 monclient(hunting): picked mon.noname-a con 0x7f75541430c0 addr v1:172.21.15.83:6789/0 2024-03-25T19:21:38.538 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.537+0000 7f7559e93640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7554142520 con 0x7f75541430c0 2024-03-25T19:21:38.538 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.537+0000 7f7559e93640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75540b28a0 con 0x7f75540a98f0 2024-03-25T19:21:38.538 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.537+0000 7f7559e93640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75540b2a30 con 0x7f755413f710 2024-03-25T19:21:38.538 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.537+0000 7f7559e93640 10 monclient(hunting): _renew_subs 2024-03-25T19:21:38.538 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.537+0000 7f7559e93640 10 monclient(hunting): authenticate will time out at 6155.808986s 2024-03-25T19:21:38.550 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.551+0000 7f7553fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f75541430c0 0x7f755414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:51574/0 (socket says 172.21.15.83:51574) 2024-03-25T19:21:38.550 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.551+0000 7f7553fff640 1 -- 172.21.15.83:0/1400196204 learned_addr learned my addr 172.21.15.83:0/1400196204 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:21:38.566 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.566+0000 7f7552ffd640 1 -- 172.21.15.83:0/1400196204 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3305618314 0 0) 0x7f7554142520 con 0x7f75541430c0 2024-03-25T19:21:38.566 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.566+0000 7f7552ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:21:38.566 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.566+0000 7f7552ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:38.566 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.566+0000 7f7552ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:38.566 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.566+0000 7f7552ffd640 10 monclient(hunting): my global_id is 4302 2024-03-25T19:21:38.566 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.566+0000 7f7552ffd640 1 -- 172.21.15.83:0/1400196204 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7540003520 con 0x7f75541430c0 2024-03-25T19:21:38.570 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.570+0000 7f7552ffd640 1 -- 172.21.15.83:0/1400196204 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 535924199 0 0) 0x7f75540b28a0 con 0x7f75540a98f0 2024-03-25T19:21:38.570 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.570+0000 7f7552ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:21:38.570 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.570+0000 7f7552ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:38.570 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.570+0000 7f7552ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:38.570 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.570+0000 7f7552ffd640 10 monclient(hunting): my global_id is 4306 2024-03-25T19:21:38.570 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.570+0000 7f7552ffd640 1 -- 172.21.15.83:0/1400196204 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7554142520 con 0x7f75540a98f0 2024-03-25T19:21:38.576 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.576+0000 7f7552ffd640 1 -- 172.21.15.83:0/1400196204 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 220132084 0 0) 0x7f75540b2a30 con 0x7f755413f710 2024-03-25T19:21:38.576 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.576+0000 7f7552ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:21:38.576 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.576+0000 7f7552ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:38.576 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.576+0000 7f7552ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:38.576 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.576+0000 7f7552ffd640 10 monclient(hunting): my global_id is 4313 2024-03-25T19:21:38.576 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.576+0000 7f7552ffd640 1 -- 172.21.15.83:0/1400196204 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75540b28a0 con 0x7f755413f710 2024-03-25T19:21:38.577 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.577+0000 7f7552ffd640 1 -- 172.21.15.83:0/1400196204 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3291161175 0 0) 0x7f7540003520 con 0x7f75541430c0 2024-03-25T19:21:38.577 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.577+0000 7f7552ffd640 1 -- 172.21.15.83:0/1400196204 >> v1:172.21.15.83:6790/0 conn(0x7f755413f710 legacy=0x7f75541429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:38.577 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.577+0000 7f7552ffd640 1 -- 172.21.15.83:0/1400196204 >> v1:172.21.15.103:6789/0 conn(0x7f75540a98f0 legacy=0x7f75540a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:38.577 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.577+0000 7f7552ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:21:38.577 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.577+0000 7f7552ffd640 1 monclient: found mon.noname-a 2024-03-25T19:21:38.577 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.577+0000 7f7552ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:21:38.578 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.577+0000 7f7552ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.83:6789/0 2024-03-25T19:21:38.578 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.577+0000 7f7552ffd640 1 -- 172.21.15.83:0/1400196204 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75540b44c0 con 0x7f75541430c0 2024-03-25T19:21:38.578 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.577+0000 7f7552ffd640 10 monclient: _finish_auth 0 2024-03-25T19:21:38.578 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.577+0000 7f7552ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:21:38.578 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.577+0000 7f7552ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:21:38.578 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.577+0000 7f7552ffd640 1 -- 172.21.15.83:0/1400196204 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f754c001a10 con 0x7f75541430c0 2024-03-25T19:21:38.578 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.577+0000 7f7559e93640 5 monclient: authenticate success, global_id 4302 2024-03-25T19:21:38.578 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.577+0000 7f7559e93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:21:38.578 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.577+0000 7f7552ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:38.578 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.577+0000 7f7552ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:21:38.578 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.577+0000 7f7552ffd640 10 monclient: dump: 2024-03-25T19:21:38.578 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:21:38.578 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:21:38.578 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:38.578 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:38.579 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:38.579 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:21:38.579 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:21:38.579 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:21:38.579 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:21:38.579 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:38.579 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.577+0000 7f7559e93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:21:38.582 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.582+0000 7f7552ffd640 1 -- 172.21.15.83:0/1400196204 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f754c001bf0 con 0x7f75541430c0 2024-03-25T19:21:38.582 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.582+0000 7f7552ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:21:38.582 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.582+0000 7f7559e93640 10 monclient: get_monmap_and_config success 2024-03-25T19:21:38.582 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.582+0000 7f7559e93640 4 set_mon_vals no callback set 2024-03-25T19:21:38.583 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.583+0000 7f7559e93640 10 monclient: shutdown 2024-03-25T19:21:38.583 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.583+0000 7f7559e93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:21:38.583 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.583+0000 7f7559e93640 1 -- 172.21.15.83:0/1400196204 >> v1:172.21.15.83:6789/0 conn(0x7f75541430c0 legacy=0x7f755414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:38.583 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.583+0000 7f7559e93640 1 -- 172.21.15.83:0/1400196204 shutdown_connections 2024-03-25T19:21:38.583 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.583+0000 7f7559e93640 1 -- 172.21.15.83:0/1400196204 >> 172.21.15.83:0/1400196204 conn(0x7f75540b7ea0 msgr2=0x7f75540b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:21:38.583 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.583+0000 7f7559e93640 1 -- 172.21.15.83:0/1400196204 shutdown_connections 2024-03-25T19:21:38.583 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.583+0000 7f7559e93640 1 -- 172.21.15.83:0/1400196204 wait complete. 2024-03-25T19:21:38.584 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.584+0000 7f7559e93640 10 monclient: build_initial_monmap 2024-03-25T19:21:38.584 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.584+0000 7f7559e93640 10 monclient: monmap: 2024-03-25T19:21:38.584 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:21:38.584 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:21:38.584 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:21:38.584 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:21:38.584 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:21:38.584 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:21:38.584 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:21:38.584 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:21:38.584 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:21:38.584 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:38.585 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.585+0000 7f7559e93640 1 librados: starting msgr at 2024-03-25T19:21:38.586 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.585+0000 7f7559e93640 1 librados: starting objecter 2024-03-25T19:21:38.586 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.585+0000 7f7559e93640 1 Processor -- start 2024-03-25T19:21:38.586 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.585+0000 7f7559e93640 1 -- start start 2024-03-25T19:21:38.586 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.585+0000 7f7559e93640 1 librados: setting wanted keys 2024-03-25T19:21:38.586 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.585+0000 7f7559e93640 1 librados: calling monclient init 2024-03-25T19:21:38.586 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.585+0000 7f7559e93640 10 monclient: init 2024-03-25T19:21:38.586 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.585+0000 7f7559e93640 10 monclient: _reopen_session rank -1 2024-03-25T19:21:38.586 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.585+0000 7f7559e93640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:21:38.586 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.585+0000 7f7559e93640 10 monclient(hunting): picked mon.noname-c con 0x7f75540a98f0 addr v1:172.21.15.83:6790/0 2024-03-25T19:21:38.586 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.585+0000 7f7559e93640 10 monclient(hunting): picked mon.noname-a con 0x7f755413f710 addr v1:172.21.15.83:6789/0 2024-03-25T19:21:38.586 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.585+0000 7f7559e93640 10 monclient(hunting): picked mon.noname-b con 0x7f75541430c0 addr v1:172.21.15.103:6789/0 2024-03-25T19:21:38.586 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.585+0000 7f7559e93640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75541db7c0 con 0x7f755413f710 2024-03-25T19:21:38.586 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.585+0000 7f7559e93640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75541dab90 con 0x7f75541430c0 2024-03-25T19:21:38.586 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.585+0000 7f7559e93640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75541398b0 con 0x7f75540a98f0 2024-03-25T19:21:38.586 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.585+0000 7f7559e93640 10 monclient(hunting): _renew_subs 2024-03-25T19:21:38.586 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.585+0000 7f7559e93640 10 monclient(hunting): authenticate will time out at 6155.857603s 2024-03-25T19:21:38.595 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.595+0000 7f754bfff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f755413f710 0x7f7554133ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:51620/0 (socket says 172.21.15.83:51620) 2024-03-25T19:21:38.595 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.595+0000 7f754bfff640 1 -- 172.21.15.83:0/2895821202 learned_addr learned my addr 172.21.15.83:0/2895821202 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:21:38.598 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.599+0000 7f7550ff9640 10 client.?.objecter ms_handle_connect 0x7f755413f710 2024-03-25T19:21:38.598 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.599+0000 7f7550ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:38.606 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.607+0000 7f7550ff9640 10 client.?.objecter ms_handle_connect 0x7f75541430c0 2024-03-25T19:21:38.606 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.607+0000 7f7550ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:38.610 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.610+0000 7f7550ff9640 10 client.?.objecter ms_handle_connect 0x7f75540a98f0 2024-03-25T19:21:38.610 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.610+0000 7f7550ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:38.613 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.614+0000 7f7550ff9640 1 -- 172.21.15.83:0/2895821202 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4017155294 0 0) 0x7f75541dab90 con 0x7f75541430c0 2024-03-25T19:21:38.614 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.614+0000 7f7550ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:21:38.614 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.614+0000 7f7550ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:38.614 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.614+0000 7f7550ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:38.614 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.614+0000 7f7550ff9640 10 monclient(hunting): my global_id is 4309 2024-03-25T19:21:38.614 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.614+0000 7f7550ff9640 1 -- 172.21.15.83:0/2895821202 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7534004eb0 con 0x7f75541430c0 2024-03-25T19:21:38.614 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.615+0000 7f7550ff9640 1 -- 172.21.15.83:0/2895821202 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3767206268 0 0) 0x7f75541db7c0 con 0x7f755413f710 2024-03-25T19:21:38.614 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.615+0000 7f7550ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:21:38.614 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.615+0000 7f7550ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:38.614 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.615+0000 7f7550ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:38.615 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.615+0000 7f7550ff9640 10 monclient(hunting): my global_id is 4305 2024-03-25T19:21:38.615 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.615+0000 7f7550ff9640 1 -- 172.21.15.83:0/2895821202 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75541dab90 con 0x7f755413f710 2024-03-25T19:21:38.624 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.625+0000 7f7550ff9640 1 -- 172.21.15.83:0/2895821202 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2754855177 0 0) 0x7f75541398b0 con 0x7f75540a98f0 2024-03-25T19:21:38.624 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.625+0000 7f7550ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:21:38.624 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.625+0000 7f7550ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:38.624 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.625+0000 7f7550ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:38.625 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.625+0000 7f7550ff9640 10 monclient(hunting): my global_id is 4316 2024-03-25T19:21:38.625 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.625+0000 7f7550ff9640 1 -- 172.21.15.83:0/2895821202 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75541db7c0 con 0x7f75540a98f0 2024-03-25T19:21:38.631 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.631+0000 7f7550ff9640 1 -- 172.21.15.83:0/2895821202 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3187609 0 0) 0x7f7534004eb0 con 0x7f75541430c0 2024-03-25T19:21:38.631 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.631+0000 7f7550ff9640 1 -- 172.21.15.83:0/2895821202 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75541398b0 con 0x7f75541430c0 2024-03-25T19:21:38.631 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.631+0000 7f7550ff9640 1 -- 172.21.15.83:0/2895821202 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f754c001f00 con 0x7f75541430c0 2024-03-25T19:21:38.631 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.631+0000 7f7550ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:38.631 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.631+0000 7f7550ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:21:38.631 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.631+0000 7f7550ff9640 10 monclient(hunting): dump: 2024-03-25T19:21:38.631 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:21:38.631 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:21:38.632 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:38.632 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:38.632 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:38.632 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:21:38.632 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:21:38.632 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:21:38.632 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:21:38.632 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:38.632 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.631+0000 7f7550ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:21:38.637 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.637+0000 7f7550ff9640 1 -- 172.21.15.83:0/2895821202 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2587765986 0 0) 0x7f75541dab90 con 0x7f755413f710 2024-03-25T19:21:38.637 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.637+0000 7f7550ff9640 1 -- 172.21.15.83:0/2895821202 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7534004eb0 con 0x7f755413f710 2024-03-25T19:21:38.637 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.637+0000 7f7550ff9640 1 -- 172.21.15.83:0/2895821202 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f753c0019f0 con 0x7f755413f710 2024-03-25T19:21:38.637 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.637+0000 7f7550ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:38.638 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.638+0000 7f7550ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:21:38.638 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.638+0000 7f7550ff9640 10 monclient(hunting): dump: 2024-03-25T19:21:38.638 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:21:38.638 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:21:38.638 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:38.638 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:38.638 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:38.638 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:21:38.638 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:21:38.638 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:21:38.638 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:21:38.638 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:38.648 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.649+0000 7f7550ff9640 1 -- 172.21.15.83:0/2895821202 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4220192409 0 0) 0x7f75541db7c0 con 0x7f75540a98f0 2024-03-25T19:21:38.648 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.649+0000 7f7550ff9640 1 -- 172.21.15.83:0/2895821202 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75541dab90 con 0x7f75540a98f0 2024-03-25T19:21:38.648 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.649+0000 7f7550ff9640 1 -- 172.21.15.83:0/2895821202 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f7544001870 con 0x7f75540a98f0 2024-03-25T19:21:38.649 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.649+0000 7f7550ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:38.649 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.649+0000 7f7550ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:21:38.649 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.649+0000 7f7550ff9640 10 monclient(hunting): dump: 2024-03-25T19:21:38.649 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:21:38.649 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:21:38.649 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:38.649 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:38.649 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:38.649 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:21:38.649 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:21:38.649 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:21:38.649 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:21:38.649 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:38.653 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.653+0000 7f7550ff9640 1 -- 172.21.15.83:0/2895821202 <== mon.1 v1:172.21.15.103:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3839034068 0 0) 0x7f75541398b0 con 0x7f75541430c0 2024-03-25T19:21:38.653 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.653+0000 7f7550ff9640 1 -- 172.21.15.83:0/2895821202 >> v1:172.21.15.83:6790/0 conn(0x7f75540a98f0 legacy=0x7f75541db0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:38.653 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.653+0000 7f7550ff9640 1 -- 172.21.15.83:0/2895821202 >> v1:172.21.15.83:6789/0 conn(0x7f755413f710 legacy=0x7f7554133ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:38.653 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.653+0000 7f7550ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:21:38.653 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.653+0000 7f7550ff9640 1 monclient: found mon.b 2024-03-25T19:21:38.653 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.653+0000 7f7550ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:21:38.653 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.653+0000 7f7550ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:21:38.653 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.653+0000 7f7550ff9640 1 -- 172.21.15.83:0/2895821202 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7554139a40 con 0x7f75541430c0 2024-03-25T19:21:38.653 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.653+0000 7f7550ff9640 10 monclient: _finish_auth 0 2024-03-25T19:21:38.653 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.653+0000 7f7550ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:21:38.653 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.653+0000 7f7550ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:21:38.653 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.653+0000 7f7559e93640 5 monclient: authenticate success, global_id 4309 2024-03-25T19:21:38.654 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.653+0000 7f7559e93640 10 monclient: _renew_subs 2024-03-25T19:21:38.654 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.653+0000 7f7559e93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:21:38.654 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.653+0000 7f7559e93640 1 -- 172.21.15.83:0/2895821202 --> v1:172.21.15.103:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7554135490 con 0x7f75541430c0 2024-03-25T19:21:38.654 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.653+0000 7f7559e93640 10 client.4309.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:21:38.654 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.653+0000 7f7559e93640 10 monclient: _renew_subs 2024-03-25T19:21:38.654 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.653+0000 7f7559e93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:21:38.654 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.653+0000 7f7559e93640 1 -- 172.21.15.83:0/2895821202 --> v1:172.21.15.103:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f75541380e0 con 0x7f75541430c0 2024-03-25T19:21:38.654 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.653+0000 7f7559e93640 1 librados: init done 2024-03-25T19:21:38.655 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.654+0000 7f7559e93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:38.655 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.654+0000 7f7559e93640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:38.655 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.654+0000 7f7559e93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:21:38.655 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.654+0000 7f7559e93640 1 -- 172.21.15.83:0/2895821202 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7554074cc0 con 0x7f75541430c0 2024-03-25T19:21:38.665 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.665+0000 7f7550ff9640 1 -- 172.21.15.83:0/2895821202 <== mon.1 v1:172.21.15.103:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f754c001590 con 0x7f75541430c0 2024-03-25T19:21:38.665 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.665+0000 7f7550ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:21:38.665 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.665+0000 7f75527fc640 4 set_mon_vals no callback set 2024-03-25T19:21:38.945 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.945+0000 7f7550ff9640 1 -- 172.21.15.83:0/2895821202 <== mon.1 v1:172.21.15.103:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f754c004fb0 con 0x7f75541430c0 2024-03-25T19:21:38.945 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.945+0000 7f7550ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:38.945 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.945+0000 7f7550ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:21:38.945 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.945+0000 7f7550ff9640 10 monclient: dump: 2024-03-25T19:21:38.945 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:21:38.945 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:21:38.945 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:38.945 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:38.945 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:38.945 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:21:38.945 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:21:38.945 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:21:38.945 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:21:38.946 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:38.946 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.946+0000 7f7550ff9640 1 -- 172.21.15.83:0/2895821202 <== mon.1 v1:172.21.15.103:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7f754c01af80 con 0x7f75541430c0 2024-03-25T19:21:38.946 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.946+0000 7f75537fe640 10 monclient: get_auth_request con 0x7f753406e0b0 auth_method 0 2024-03-25T19:21:38.946 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.947+0000 7f7550ff9640 10 client.4309.objecter ms_handle_connect 0x7f753406e0b0 2024-03-25T19:21:38.948 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.948+0000 7f7550ff9640 1 -- 172.21.15.83:0/2895821202 <== mon.1 v1:172.21.15.103:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7f754c054c10 con 0x7f75541430c0 2024-03-25T19:21:38.948 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.948+0000 7f7550ff9640 10 client.4309.objecter ms_dispatch 0x7f7554000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:21:38.948 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.948+0000 7f7550ff9640 3 client.4309.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:21:38.948 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.948+0000 7f7550ff9640 3 client.4309.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:21:38.948 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.949+0000 7f7550ff9640 20 client.4309.objecter dump_active .. 0 homeless 2024-03-25T19:21:38.951 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.952+0000 7f7550ff9640 1 -- 172.21.15.83:0/2895821202 <== mon.1 v1:172.21.15.103:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f754c050550 con 0x7f75541430c0 2024-03-25T19:21:38.951 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.952+0000 7f7550ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:38.951 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:38.952+0000 7f7550ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:21:39.186 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:39.186+0000 7f7559e93640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:39.186 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:39.186+0000 7f7559e93640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:39.186 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:39.186+0000 7f7559e93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:21:39.187 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:39.187+0000 7f7559e93640 1 -- 172.21.15.83:0/2895821202 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f75540ac310 con 0x7f75541430c0 2024-03-25T19:21:39.302 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:39.302+0000 7f7550ff9640 1 -- 172.21.15.83:0/2895821202 <== mon.1 v1:172.21.15.103:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7f754c052650 con 0x7f75541430c0 2024-03-25T19:21:39.302 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:39.302+0000 7f7550ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:39.302 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:39.302+0000 7f7550ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:21:39.302 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:21:39.303 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:21:39.305 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:39.305+0000 7f7559e93640 10 librados: watch_flush enter 2024-03-25T19:21:39.305 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:39.305+0000 7f7559e93640 10 librados: watch_flush exit 2024-03-25T19:21:39.305 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:39.305+0000 7f7559e93640 20 client.4309.objecter shutdown clearing up homeless session... 2024-03-25T19:21:39.305 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:39.305+0000 7f7559e93640 10 client.4309.objecter successfully canceled tick 2024-03-25T19:21:39.305 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:39.305+0000 7f7559e93640 1 -- 172.21.15.83:0/2895821202 >> v1:172.21.15.103:6800/345072327 conn(0x7f753406e0b0 legacy=0x7f7534070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:39.306 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:39.305+0000 7f7559e93640 10 monclient: shutdown 2024-03-25T19:21:39.306 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:39.305+0000 7f7559e93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:21:39.306 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:39.305+0000 7f7559e93640 1 -- 172.21.15.83:0/2895821202 >> v1:172.21.15.103:6789/0 conn(0x7f75541430c0 legacy=0x7f75541e2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:39.306 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:39.305+0000 7f7559e93640 1 -- 172.21.15.83:0/2895821202 shutdown_connections 2024-03-25T19:21:39.306 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:39.305+0000 7f7559e93640 1 -- 172.21.15.83:0/2895821202 >> 172.21.15.83:0/2895821202 conn(0x7f75540b7ea0 msgr2=0x7f7554074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:21:39.306 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:39.305+0000 7f7559e93640 1 -- 172.21.15.83:0/2895821202 shutdown_connections 2024-03-25T19:21:39.306 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:39.305+0000 7f7559e93640 1 -- 172.21.15.83:0/2895821202 wait complete. 2024-03-25T19:21:39.306 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:39.305+0000 7f7559e93640 1 librados: shutdown 2024-03-25T19:21:39.317 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:21:42.424 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:02:24.351 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:42.428 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:02:24.355 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:42.456 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:02:24.383 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:42.471 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:02:24.381 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:42.482 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:02:24.393 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:42.511 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:02:24.438 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:42.514 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:02:24.424 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:42.516 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:02:24.443 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:42.556 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:02:24.484 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:46.123 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:21:46.240 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.240+0000 7f942cb39640 10 monclient: get_monmap_and_config 2024-03-25T19:21:46.240 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.240+0000 7f942cb39640 10 monclient: build_initial_monmap 2024-03-25T19:21:46.240 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.240+0000 7f942cb39640 10 monclient: monmap: 2024-03-25T19:21:46.240 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:21:46.240 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:21:46.240 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:21:46.241546+0000 2024-03-25T19:21:46.240 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:21:46.241546+0000 2024-03-25T19:21:46.240 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:21:46.241 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:21:46.241 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:21:46.241 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:21:46.241 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:21:46.241 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:46.242 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.241+0000 7f942cb39640 1 Processor -- start 2024-03-25T19:21:46.242 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.241+0000 7f942cb39640 1 -- start start 2024-03-25T19:21:46.243 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.241+0000 7f942cb39640 10 monclient: init 2024-03-25T19:21:46.243 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.242+0000 7f942cb39640 10 monclient: _reopen_session rank -1 2024-03-25T19:21:46.243 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.242+0000 7f942cb39640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:21:46.243 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.242+0000 7f942cb39640 10 monclient(hunting): picked mon.noname-b con 0x7f94280a98f0 addr v1:172.21.15.83:6790/0 2024-03-25T19:21:46.243 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.242+0000 7f942cb39640 10 monclient(hunting): picked mon.noname-a con 0x7f942813f710 addr v1:172.21.15.83:6789/0 2024-03-25T19:21:46.243 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.242+0000 7f942cb39640 10 monclient(hunting): picked mon.noname-c con 0x7f94281430c0 addr v1:172.21.15.103:6789/0 2024-03-25T19:21:46.243 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.242+0000 7f942cb39640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9428142520 con 0x7f942813f710 2024-03-25T19:21:46.243 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.242+0000 7f942cb39640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94280b28a0 con 0x7f94281430c0 2024-03-25T19:21:46.243 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.242+0000 7f942cb39640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94280b2a30 con 0x7f94280a98f0 2024-03-25T19:21:46.243 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.242+0000 7f942cb39640 10 monclient(hunting): _renew_subs 2024-03-25T19:21:46.243 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.242+0000 7f942cb39640 10 monclient(hunting): authenticate will time out at 6163.514128s 2024-03-25T19:21:46.254 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.255+0000 7f9425d9b640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f942813f710 0x7f94281429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:50832/0 (socket says 172.21.15.83:50832) 2024-03-25T19:21:46.255 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.255+0000 7f9425d9b640 1 -- 172.21.15.83:0/295533843 learned_addr learned my addr 172.21.15.83:0/295533843 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:21:46.276 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.277+0000 7f942559a640 1 -- 172.21.15.83:0/295533843 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2025760635 0 0) 0x7f94280b2a30 con 0x7f94280a98f0 2024-03-25T19:21:46.276 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.277+0000 7f942559a640 10 monclient(hunting): _negotiate 2024-03-25T19:21:46.277 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.277+0000 7f942559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:46.277 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.277+0000 7f942559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:46.277 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.277+0000 7f942559a640 10 monclient(hunting): my global_id is 4319 2024-03-25T19:21:46.277 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.277+0000 7f942559a640 1 -- 172.21.15.83:0/295533843 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9410003520 con 0x7f94280a98f0 2024-03-25T19:21:46.278 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.278+0000 7f942559a640 1 -- 172.21.15.83:0/295533843 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1666974231 0 0) 0x7f94280b28a0 con 0x7f94281430c0 2024-03-25T19:21:46.278 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.278+0000 7f942559a640 10 monclient(hunting): _negotiate 2024-03-25T19:21:46.278 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.278+0000 7f942559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:46.278 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.278+0000 7f942559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:46.278 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.278+0000 7f942559a640 10 monclient(hunting): my global_id is 4312 2024-03-25T19:21:46.278 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.278+0000 7f942559a640 1 -- 172.21.15.83:0/295533843 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94280b2a30 con 0x7f94281430c0 2024-03-25T19:21:46.286 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.287+0000 7f942559a640 1 -- 172.21.15.83:0/295533843 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3897048460 0 0) 0x7f9428142520 con 0x7f942813f710 2024-03-25T19:21:46.286 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.287+0000 7f942559a640 10 monclient(hunting): _negotiate 2024-03-25T19:21:46.286 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.287+0000 7f942559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:46.287 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.287+0000 7f942559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:46.287 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.287+0000 7f942559a640 10 monclient(hunting): my global_id is 4308 2024-03-25T19:21:46.287 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.287+0000 7f942559a640 1 -- 172.21.15.83:0/295533843 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94280b28a0 con 0x7f942813f710 2024-03-25T19:21:46.290 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.290+0000 7f942559a640 1 -- 172.21.15.83:0/295533843 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3926041925 0 0) 0x7f94280b2a30 con 0x7f94281430c0 2024-03-25T19:21:46.290 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.290+0000 7f942559a640 1 -- 172.21.15.83:0/295533843 >> v1:172.21.15.83:6790/0 conn(0x7f94280a98f0 legacy=0x7f94280a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:46.290 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.290+0000 7f942559a640 1 -- 172.21.15.83:0/295533843 >> v1:172.21.15.83:6789/0 conn(0x7f942813f710 legacy=0x7f94281429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:46.290 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.290+0000 7f942559a640 10 monclient: _finish_hunting 0 2024-03-25T19:21:46.290 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.290+0000 7f942559a640 1 monclient: found mon.noname-c 2024-03-25T19:21:46.290 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.290+0000 7f942559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:21:46.290 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.290+0000 7f942559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.103:6789/0 2024-03-25T19:21:46.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.290+0000 7f942559a640 1 -- 172.21.15.83:0/295533843 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94280b44c0 con 0x7f94281430c0 2024-03-25T19:21:46.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.290+0000 7f942559a640 10 monclient: _finish_auth 0 2024-03-25T19:21:46.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.290+0000 7f942559a640 10 monclient: _check_auth_tickets 2024-03-25T19:21:46.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.290+0000 7f942559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:21:46.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.290+0000 7f942559a640 1 -- 172.21.15.83:0/295533843 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f94200019f0 con 0x7f94281430c0 2024-03-25T19:21:46.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.290+0000 7f942559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:46.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.290+0000 7f942559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:21:46.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.290+0000 7f942559a640 10 monclient: dump: 2024-03-25T19:21:46.291 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:21:46.291 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:21:46.291 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:46.291 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:46.291 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:46.291 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:21:46.291 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:21:46.292 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:21:46.292 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:21:46.292 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:46.292 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.290+0000 7f942cb39640 5 monclient: authenticate success, global_id 4312 2024-03-25T19:21:46.292 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.290+0000 7f942cb39640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:21:46.307 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.307+0000 7f942559a640 1 -- 172.21.15.83:0/295533843 <== mon.1 v1:172.21.15.103:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9420001b50 con 0x7f94281430c0 2024-03-25T19:21:46.307 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.307+0000 7f942559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:21:46.307 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.307+0000 7f942559a640 1 -- 172.21.15.83:0/295533843 <== mon.1 v1:172.21.15.103:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f9420001f60 con 0x7f94281430c0 2024-03-25T19:21:46.307 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.307+0000 7f942cb39640 10 monclient: get_monmap_and_config success 2024-03-25T19:21:46.307 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.307+0000 7f942cb39640 4 set_mon_vals no callback set 2024-03-25T19:21:46.307 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.307+0000 7f942559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:46.307 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.307+0000 7f942559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:21:46.307 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.307+0000 7f942559a640 10 monclient: dump: 2024-03-25T19:21:46.307 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:21:46.307 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:21:46.307 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:46.308 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:46.308 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:46.308 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:21:46.308 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:21:46.308 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:21:46.308 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:21:46.308 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:46.309 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.308+0000 7f942cb39640 10 monclient: shutdown 2024-03-25T19:21:46.309 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.308+0000 7f942cb39640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:21:46.309 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.308+0000 7f942cb39640 1 -- 172.21.15.83:0/295533843 >> v1:172.21.15.103:6789/0 conn(0x7f94281430c0 legacy=0x7f942814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:46.309 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.308+0000 7f942cb39640 1 -- 172.21.15.83:0/295533843 shutdown_connections 2024-03-25T19:21:46.310 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.308+0000 7f942cb39640 1 -- 172.21.15.83:0/295533843 >> 172.21.15.83:0/295533843 conn(0x7f94280b7ea0 msgr2=0x7f94280b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:21:46.310 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.308+0000 7f942cb39640 1 -- 172.21.15.83:0/295533843 shutdown_connections 2024-03-25T19:21:46.310 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.308+0000 7f942cb39640 1 -- 172.21.15.83:0/295533843 wait complete. 2024-03-25T19:21:46.310 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.309+0000 7f942cb39640 10 monclient: build_initial_monmap 2024-03-25T19:21:46.310 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.309+0000 7f942cb39640 10 monclient: monmap: 2024-03-25T19:21:46.310 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:21:46.310 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:21:46.310 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:21:46.310 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:21:46.310 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:21:46.310 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:21:46.310 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:21:46.310 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:21:46.310 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:21:46.311 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:46.311 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.310+0000 7f942cb39640 1 librados: starting msgr at 2024-03-25T19:21:46.311 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.310+0000 7f942cb39640 1 librados: starting objecter 2024-03-25T19:21:46.311 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.310+0000 7f942cb39640 1 Processor -- start 2024-03-25T19:21:46.311 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.310+0000 7f942cb39640 1 -- start start 2024-03-25T19:21:46.311 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.310+0000 7f942cb39640 1 librados: setting wanted keys 2024-03-25T19:21:46.311 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.310+0000 7f942cb39640 1 librados: calling monclient init 2024-03-25T19:21:46.311 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.310+0000 7f942cb39640 10 monclient: init 2024-03-25T19:21:46.312 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.310+0000 7f942cb39640 10 monclient: _reopen_session rank -1 2024-03-25T19:21:46.312 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.310+0000 7f942cb39640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:21:46.312 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.310+0000 7f942cb39640 10 monclient(hunting): picked mon.noname-b con 0x7f94280a98f0 addr v1:172.21.15.103:6789/0 2024-03-25T19:21:46.312 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.310+0000 7f942cb39640 10 monclient(hunting): picked mon.noname-c con 0x7f942813f710 addr v1:172.21.15.83:6790/0 2024-03-25T19:21:46.312 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.310+0000 7f942cb39640 10 monclient(hunting): picked mon.noname-a con 0x7f94281430c0 addr v1:172.21.15.83:6789/0 2024-03-25T19:21:46.312 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.310+0000 7f942cb39640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f942813c190 con 0x7f94281430c0 2024-03-25T19:21:46.312 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.310+0000 7f942cb39640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94280b7cf0 con 0x7f94280a98f0 2024-03-25T19:21:46.312 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.310+0000 7f942cb39640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f942813ad50 con 0x7f942813f710 2024-03-25T19:21:46.312 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.310+0000 7f942cb39640 10 monclient(hunting): _renew_subs 2024-03-25T19:21:46.313 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.310+0000 7f942cb39640 10 monclient(hunting): authenticate will time out at 6163.582571s 2024-03-25T19:21:46.314 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.314+0000 7f942659c640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f94280a98f0 0x7f942813ba80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:36202/0 (socket says 172.21.15.83:36202) 2024-03-25T19:21:46.314 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.314+0000 7f942659c640 1 -- 172.21.15.83:0/226351620 learned_addr learned my addr 172.21.15.83:0/226351620 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:21:46.314 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.315+0000 7f9416ffd640 10 client.?.objecter ms_handle_connect 0x7f942813f710 2024-03-25T19:21:46.314 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.315+0000 7f9416ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:46.325 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.326+0000 7f9416ffd640 10 client.?.objecter ms_handle_connect 0x7f94280a98f0 2024-03-25T19:21:46.325 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.326+0000 7f9416ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:46.327 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.327+0000 7f9416ffd640 10 client.?.objecter ms_handle_connect 0x7f94281430c0 2024-03-25T19:21:46.327 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.327+0000 7f9416ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:46.331 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.331+0000 7f9416ffd640 1 -- 172.21.15.83:0/226351620 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1763288617 0 0) 0x7f942813ad50 con 0x7f942813f710 2024-03-25T19:21:46.331 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.331+0000 7f9416ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:21:46.331 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.331+0000 7f9416ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:46.331 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.331+0000 7f9416ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:46.331 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.331+0000 7f9416ffd640 10 monclient(hunting): my global_id is 4322 2024-03-25T19:21:46.331 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.331+0000 7f9416ffd640 1 -- 172.21.15.83:0/226351620 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9408004eb0 con 0x7f942813f710 2024-03-25T19:21:46.336 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.336+0000 7f9416ffd640 1 -- 172.21.15.83:0/226351620 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 7007305 0 0) 0x7f94280b44c0 con 0x7f94281430c0 2024-03-25T19:21:46.336 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.336+0000 7f9416ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:21:46.336 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.336+0000 7f9416ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:46.336 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.336+0000 7f9416ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:46.336 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.336+0000 7f9416ffd640 10 monclient(hunting): my global_id is 4311 2024-03-25T19:21:46.336 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.336+0000 7f9416ffd640 1 -- 172.21.15.83:0/226351620 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f942813ad50 con 0x7f94281430c0 2024-03-25T19:21:46.339 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.340+0000 7f9416ffd640 1 -- 172.21.15.83:0/226351620 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3753844576 0 0) 0x7f94280b7cf0 con 0x7f94280a98f0 2024-03-25T19:21:46.339 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.340+0000 7f9416ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:21:46.339 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.340+0000 7f9416ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:46.339 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.340+0000 7f9416ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:46.340 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.340+0000 7f9416ffd640 10 monclient(hunting): my global_id is 4315 2024-03-25T19:21:46.340 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.340+0000 7f9416ffd640 1 -- 172.21.15.83:0/226351620 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94280b44c0 con 0x7f94280a98f0 2024-03-25T19:21:46.352 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.352+0000 7f9416ffd640 1 -- 172.21.15.83:0/226351620 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3080417506 0 0) 0x7f94280b44c0 con 0x7f94280a98f0 2024-03-25T19:21:46.352 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.352+0000 7f9416ffd640 1 -- 172.21.15.83:0/226351620 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94280b7cf0 con 0x7f94280a98f0 2024-03-25T19:21:46.352 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.352+0000 7f9416ffd640 1 -- 172.21.15.83:0/226351620 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f9418001910 con 0x7f94280a98f0 2024-03-25T19:21:46.352 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.352+0000 7f9416ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:46.352 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.352+0000 7f9416ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:21:46.352 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.352+0000 7f9416ffd640 10 monclient(hunting): dump: 2024-03-25T19:21:46.352 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:21:46.352 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:21:46.352 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:46.352 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:46.353 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:46.353 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:21:46.353 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:21:46.353 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:21:46.353 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:21:46.353 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:46.353 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.352+0000 7f9416ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:21:46.353 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.353+0000 7f9416ffd640 1 -- 172.21.15.83:0/226351620 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 135271220 0 0) 0x7f942813ad50 con 0x7f94281430c0 2024-03-25T19:21:46.354 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.353+0000 7f9416ffd640 1 -- 172.21.15.83:0/226351620 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94280b44c0 con 0x7f94281430c0 2024-03-25T19:21:46.354 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.353+0000 7f9416ffd640 1 -- 172.21.15.83:0/226351620 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f9420001cc0 con 0x7f94281430c0 2024-03-25T19:21:46.354 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.353+0000 7f9416ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:46.354 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.353+0000 7f9416ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:21:46.354 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.353+0000 7f9416ffd640 10 monclient(hunting): dump: 2024-03-25T19:21:46.354 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:21:46.354 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:21:46.354 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:46.354 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:46.354 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:46.354 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:21:46.354 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:21:46.355 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:21:46.355 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:21:46.355 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:46.358 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.358+0000 7f9416ffd640 1 -- 172.21.15.83:0/226351620 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3764213493 0 0) 0x7f9408004eb0 con 0x7f942813f710 2024-03-25T19:21:46.358 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.358+0000 7f9416ffd640 1 -- 172.21.15.83:0/226351620 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f942813ad50 con 0x7f942813f710 2024-03-25T19:21:46.358 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.358+0000 7f9416ffd640 1 -- 172.21.15.83:0/226351620 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f941c0018f0 con 0x7f942813f710 2024-03-25T19:21:46.358 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.358+0000 7f9416ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:46.358 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.358+0000 7f9416ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:21:46.358 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.358+0000 7f9416ffd640 10 monclient(hunting): dump: 2024-03-25T19:21:46.358 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:21:46.358 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:21:46.358 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:46.358 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:46.359 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:46.359 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:21:46.359 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:21:46.359 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:21:46.359 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:21:46.359 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:46.364 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.364+0000 7f9416ffd640 1 -- 172.21.15.83:0/226351620 <== mon.1 v1:172.21.15.103:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3944323265 0 0) 0x7f94280b7cf0 con 0x7f94280a98f0 2024-03-25T19:21:46.364 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.364+0000 7f9416ffd640 1 -- 172.21.15.83:0/226351620 >> v1:172.21.15.83:6790/0 conn(0x7f942813f710 legacy=0x7f94281df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:46.364 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.364+0000 7f9416ffd640 1 -- 172.21.15.83:0/226351620 >> v1:172.21.15.83:6789/0 conn(0x7f94281430c0 legacy=0x7f94281e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:46.364 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.364+0000 7f9416ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:21:46.365 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.364+0000 7f9416ffd640 1 monclient: found mon.b 2024-03-25T19:21:46.365 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.364+0000 7f9416ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:21:46.365 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.364+0000 7f9416ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:21:46.365 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.364+0000 7f9416ffd640 1 -- 172.21.15.83:0/226351620 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f942813afc0 con 0x7f94280a98f0 2024-03-25T19:21:46.365 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.364+0000 7f9416ffd640 10 monclient: _finish_auth 0 2024-03-25T19:21:46.365 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.364+0000 7f9416ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:21:46.365 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.364+0000 7f9416ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:21:46.365 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.364+0000 7f942cb39640 5 monclient: authenticate success, global_id 4315 2024-03-25T19:21:46.365 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.364+0000 7f942cb39640 10 monclient: _renew_subs 2024-03-25T19:21:46.365 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.364+0000 7f942cb39640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:21:46.365 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.364+0000 7f942cb39640 1 -- 172.21.15.83:0/226351620 --> v1:172.21.15.103:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f942813b2a0 con 0x7f94280a98f0 2024-03-25T19:21:46.365 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.364+0000 7f942cb39640 10 client.4315.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:21:46.365 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.364+0000 7f942cb39640 10 monclient: _renew_subs 2024-03-25T19:21:46.365 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.364+0000 7f942cb39640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:21:46.365 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.364+0000 7f942cb39640 1 -- 172.21.15.83:0/226351620 --> v1:172.21.15.103:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f942812fa90 con 0x7f94280a98f0 2024-03-25T19:21:46.366 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.364+0000 7f942cb39640 1 librados: init done 2024-03-25T19:21:46.366 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.365+0000 7f942cb39640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:46.366 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.365+0000 7f942cb39640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:46.366 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.365+0000 7f942cb39640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:21:46.366 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.365+0000 7f942cb39640 1 -- 172.21.15.83:0/226351620 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94280b3ef0 con 0x7f94280a98f0 2024-03-25T19:21:46.674 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.674+0000 7f9416ffd640 1 -- 172.21.15.83:0/226351620 <== mon.1 v1:172.21.15.103:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9418003610 con 0x7f94280a98f0 2024-03-25T19:21:46.674 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.674+0000 7f9416ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:21:46.674 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.674+0000 7f9416ffd640 1 -- 172.21.15.83:0/226351620 <== mon.1 v1:172.21.15.103:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f94180039e0 con 0x7f94280a98f0 2024-03-25T19:21:46.674 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.674+0000 7f9416ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:46.674 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.674+0000 7f9416ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:21:46.674 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.674+0000 7f9416ffd640 10 monclient: dump: 2024-03-25T19:21:46.674 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:21:46.674 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:21:46.674 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:46.674 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:46.674 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:46.675 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:21:46.675 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:21:46.675 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:21:46.675 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:21:46.675 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:46.675 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.674+0000 7f9424d99640 4 set_mon_vals no callback set 2024-03-25T19:21:46.675 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.675+0000 7f9416ffd640 1 -- 172.21.15.83:0/226351620 <== mon.1 v1:172.21.15.103:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7f9418003bc0 con 0x7f94280a98f0 2024-03-25T19:21:46.675 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.675+0000 7f9425d9b640 10 monclient: get_auth_request con 0x7f940806e0f0 auth_method 0 2024-03-25T19:21:46.676 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.676+0000 7f9416ffd640 10 client.4315.objecter ms_handle_connect 0x7f940806e0f0 2024-03-25T19:21:46.680 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.680+0000 7f9416ffd640 1 -- 172.21.15.83:0/226351620 <== mon.1 v1:172.21.15.103:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7f94180547a0 con 0x7f94280a98f0 2024-03-25T19:21:46.680 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.680+0000 7f9416ffd640 10 client.4315.objecter ms_dispatch 0x7f9428000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:21:46.680 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.680+0000 7f9416ffd640 3 client.4315.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:21:46.680 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.680+0000 7f9416ffd640 3 client.4315.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:21:46.680 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.680+0000 7f9416ffd640 20 client.4315.objecter dump_active .. 0 homeless 2024-03-25T19:21:46.681 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.681+0000 7f9416ffd640 1 -- 172.21.15.83:0/226351620 <== mon.1 v1:172.21.15.103:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f94180502b0 con 0x7f94280a98f0 2024-03-25T19:21:46.681 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.681+0000 7f9416ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:46.681 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.681+0000 7f9416ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:21:46.930 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.930+0000 7f942cb39640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:46.930 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.931+0000 7f942cb39640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:46.930 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.931+0000 7f942cb39640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:21:46.930 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.931+0000 7f942cb39640 1 -- 172.21.15.83:0/226351620 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94280627b0 con 0x7f94280a98f0 2024-03-25T19:21:46.956 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.956+0000 7f9416ffd640 1 -- 172.21.15.83:0/226351620 <== mon.1 v1:172.21.15.103:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7f94180521e0 con 0x7f94280a98f0 2024-03-25T19:21:46.956 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.956+0000 7f9416ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:46.956 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.956+0000 7f9416ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:21:46.956 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:21:46.956 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:21:46.959 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.958+0000 7f942cb39640 10 librados: watch_flush enter 2024-03-25T19:21:46.959 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.958+0000 7f942cb39640 10 librados: watch_flush exit 2024-03-25T19:21:46.959 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.958+0000 7f942cb39640 20 client.4315.objecter shutdown clearing up homeless session... 2024-03-25T19:21:46.959 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.958+0000 7f942cb39640 10 client.4315.objecter successfully canceled tick 2024-03-25T19:21:46.959 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.959+0000 7f942cb39640 1 -- 172.21.15.83:0/226351620 >> v1:172.21.15.103:6800/345072327 conn(0x7f940806e0f0 legacy=0x7f94080705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:46.959 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.959+0000 7f942cb39640 10 monclient: shutdown 2024-03-25T19:21:46.959 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.959+0000 7f942cb39640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:21:46.959 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.959+0000 7f942cb39640 1 -- 172.21.15.83:0/226351620 >> v1:172.21.15.103:6789/0 conn(0x7f94280a98f0 legacy=0x7f942813ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:46.959 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.959+0000 7f942cb39640 1 -- 172.21.15.83:0/226351620 shutdown_connections 2024-03-25T19:21:46.959 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.959+0000 7f942cb39640 1 -- 172.21.15.83:0/226351620 >> 172.21.15.83:0/226351620 conn(0x7f94280b7ea0 msgr2=0x7f9428074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:21:46.960 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.959+0000 7f942cb39640 1 -- 172.21.15.83:0/226351620 shutdown_connections 2024-03-25T19:21:46.960 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.959+0000 7f942cb39640 1 -- 172.21.15.83:0/226351620 wait complete. 2024-03-25T19:21:46.960 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:46.959+0000 7f942cb39640 1 librados: shutdown 2024-03-25T19:21:46.970 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:21:53.776 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:21:53.888 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.888+0000 7feda8aae640 10 monclient: get_monmap_and_config 2024-03-25T19:21:53.888 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.888+0000 7feda8aae640 10 monclient: build_initial_monmap 2024-03-25T19:21:53.888 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.889+0000 7feda8aae640 10 monclient: monmap: 2024-03-25T19:21:53.889 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:21:53.889 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:21:53.889 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:21:53.889880+0000 2024-03-25T19:21:53.889 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:21:53.889880+0000 2024-03-25T19:21:53.889 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:21:53.889 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:21:53.889 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:21:53.889 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:21:53.889 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:21:53.889 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:53.890 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.889+0000 7feda8aae640 1 Processor -- start 2024-03-25T19:21:53.891 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.889+0000 7feda8aae640 1 -- start start 2024-03-25T19:21:53.891 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.889+0000 7feda8aae640 10 monclient: init 2024-03-25T19:21:53.891 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.890+0000 7feda8aae640 10 monclient: _reopen_session rank -1 2024-03-25T19:21:53.891 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.890+0000 7feda8aae640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:21:53.891 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.890+0000 7feda8aae640 10 monclient(hunting): picked mon.noname-b con 0x7feda40a98f0 addr v1:172.21.15.83:6790/0 2024-03-25T19:21:53.891 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.890+0000 7feda8aae640 10 monclient(hunting): picked mon.noname-a con 0x7feda413f710 addr v1:172.21.15.83:6789/0 2024-03-25T19:21:53.891 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.890+0000 7feda8aae640 10 monclient(hunting): picked mon.noname-c con 0x7feda41430c0 addr v1:172.21.15.103:6789/0 2024-03-25T19:21:53.891 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.890+0000 7feda8aae640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda4142520 con 0x7feda413f710 2024-03-25T19:21:53.891 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.890+0000 7feda8aae640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda40b28a0 con 0x7feda41430c0 2024-03-25T19:21:53.891 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.890+0000 7feda8aae640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda40b2a30 con 0x7feda40a98f0 2024-03-25T19:21:53.891 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.890+0000 7feda8aae640 10 monclient(hunting): _renew_subs 2024-03-25T19:21:53.891 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.890+0000 7feda8aae640 10 monclient(hunting): authenticate will time out at 6171.162655s 2024-03-25T19:21:53.896 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.896+0000 7feda1d9b640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7feda413f710 0x7feda41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:55546/0 (socket says 172.21.15.83:55546) 2024-03-25T19:21:53.896 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.896+0000 7feda1d9b640 1 -- 172.21.15.83:0/3648260380 learned_addr learned my addr 172.21.15.83:0/3648260380 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:21:53.909 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.909+0000 7feda159a640 1 -- 172.21.15.83:0/3648260380 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2921379675 0 0) 0x7feda4142520 con 0x7feda413f710 2024-03-25T19:21:53.909 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.909+0000 7feda159a640 10 monclient(hunting): _negotiate 2024-03-25T19:21:53.909 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.909+0000 7feda159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:53.909 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.909+0000 7feda159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:53.909 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.909+0000 7feda159a640 10 monclient(hunting): my global_id is 4314 2024-03-25T19:21:53.909 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.909+0000 7feda159a640 1 -- 172.21.15.83:0/3648260380 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed90003520 con 0x7feda413f710 2024-03-25T19:21:53.916 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.917+0000 7feda159a640 1 -- 172.21.15.83:0/3648260380 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4107627998 0 0) 0x7feda40b2a30 con 0x7feda40a98f0 2024-03-25T19:21:53.916 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.917+0000 7feda159a640 10 monclient(hunting): _negotiate 2024-03-25T19:21:53.916 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.917+0000 7feda159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:53.917 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.917+0000 7feda159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:53.917 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.917+0000 7feda159a640 10 monclient(hunting): my global_id is 4325 2024-03-25T19:21:53.917 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.917+0000 7feda159a640 1 -- 172.21.15.83:0/3648260380 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feda4142520 con 0x7feda40a98f0 2024-03-25T19:21:53.927 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.927+0000 7feda159a640 1 -- 172.21.15.83:0/3648260380 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4120069257 0 0) 0x7fed90003520 con 0x7feda413f710 2024-03-25T19:21:53.927 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.927+0000 7feda159a640 1 -- 172.21.15.83:0/3648260380 >> v1:172.21.15.83:6790/0 conn(0x7feda40a98f0 legacy=0x7feda40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:53.927 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.927+0000 7feda159a640 1 -- 172.21.15.83:0/3648260380 >> v1:172.21.15.103:6789/0 conn(0x7feda41430c0 legacy=0x7feda414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:53.927 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.927+0000 7feda159a640 10 monclient: _finish_hunting 0 2024-03-25T19:21:53.927 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.927+0000 7feda159a640 1 monclient: found mon.noname-a 2024-03-25T19:21:53.927 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.927+0000 7feda159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:21:53.927 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.927+0000 7feda159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.83:6789/0 2024-03-25T19:21:53.927 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.927+0000 7feda159a640 1 -- 172.21.15.83:0/3648260380 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feda40b44c0 con 0x7feda413f710 2024-03-25T19:21:53.928 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.927+0000 7feda159a640 10 monclient: _finish_auth 0 2024-03-25T19:21:53.928 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.927+0000 7feda159a640 10 monclient: _check_auth_tickets 2024-03-25T19:21:53.928 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.927+0000 7feda159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:21:53.928 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.927+0000 7feda159a640 1 -- 172.21.15.83:0/3648260380 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fed8c001690 con 0x7feda413f710 2024-03-25T19:21:53.928 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.927+0000 7feda8aae640 5 monclient: authenticate success, global_id 4314 2024-03-25T19:21:53.928 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.927+0000 7feda8aae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:21:53.928 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.927+0000 7feda159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:53.928 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.927+0000 7feda159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:21:53.928 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.927+0000 7feda159a640 10 monclient: dump: 2024-03-25T19:21:53.928 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:21:53.928 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:21:53.928 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:53.928 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:53.928 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:53.928 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:21:53.928 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:21:53.929 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:21:53.929 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:21:53.929 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:53.929 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.927+0000 7feda8aae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:21:53.937 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.937+0000 7feda159a640 1 -- 172.21.15.83:0/3648260380 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed8c0017f0 con 0x7feda413f710 2024-03-25T19:21:53.937 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.937+0000 7feda159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:21:53.937 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.937+0000 7feda159a640 1 -- 172.21.15.83:0/3648260380 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fed8c001c00 con 0x7feda413f710 2024-03-25T19:21:53.937 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.937+0000 7feda8aae640 10 monclient: get_monmap_and_config success 2024-03-25T19:21:53.937 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.937+0000 7feda8aae640 4 set_mon_vals no callback set 2024-03-25T19:21:53.937 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.937+0000 7feda159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:53.937 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.937+0000 7feda159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:21:53.937 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.937+0000 7feda159a640 10 monclient: dump: 2024-03-25T19:21:53.937 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:21:53.937 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:21:53.937 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:53.937 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:53.937 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:53.937 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:21:53.938 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:21:53.938 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:21:53.938 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:21:53.938 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:53.939 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.938+0000 7feda8aae640 10 monclient: shutdown 2024-03-25T19:21:53.939 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.938+0000 7feda8aae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:21:53.939 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.938+0000 7feda8aae640 1 -- 172.21.15.83:0/3648260380 >> v1:172.21.15.83:6789/0 conn(0x7feda413f710 legacy=0x7feda41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:53.939 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.938+0000 7feda8aae640 1 -- 172.21.15.83:0/3648260380 shutdown_connections 2024-03-25T19:21:53.939 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.938+0000 7feda8aae640 1 -- 172.21.15.83:0/3648260380 >> 172.21.15.83:0/3648260380 conn(0x7feda40b7ea0 msgr2=0x7feda40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:21:53.939 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.938+0000 7feda8aae640 1 -- 172.21.15.83:0/3648260380 shutdown_connections 2024-03-25T19:21:53.939 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.938+0000 7feda8aae640 1 -- 172.21.15.83:0/3648260380 wait complete. 2024-03-25T19:21:53.939 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.939+0000 7feda8aae640 10 monclient: build_initial_monmap 2024-03-25T19:21:53.940 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.939+0000 7feda8aae640 10 monclient: monmap: 2024-03-25T19:21:53.940 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:21:53.940 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:21:53.940 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:21:53.940 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:21:53.940 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:21:53.940 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:21:53.940 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:21:53.940 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:21:53.940 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:21:53.940 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:53.940 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.939+0000 7feda8aae640 1 librados: starting msgr at 2024-03-25T19:21:53.940 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.939+0000 7feda8aae640 1 librados: starting objecter 2024-03-25T19:21:53.940 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.939+0000 7feda8aae640 1 Processor -- start 2024-03-25T19:21:53.940 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.939+0000 7feda8aae640 1 -- start start 2024-03-25T19:21:53.940 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.939+0000 7feda8aae640 1 librados: setting wanted keys 2024-03-25T19:21:53.941 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.939+0000 7feda8aae640 1 librados: calling monclient init 2024-03-25T19:21:53.941 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.939+0000 7feda8aae640 10 monclient: init 2024-03-25T19:21:53.941 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.940+0000 7feda8aae640 10 monclient: _reopen_session rank -1 2024-03-25T19:21:53.942 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.940+0000 7feda8aae640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:21:53.942 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.940+0000 7feda8aae640 10 monclient(hunting): picked mon.noname-a con 0x7feda40a98f0 addr v1:172.21.15.83:6789/0 2024-03-25T19:21:53.942 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.940+0000 7feda8aae640 10 monclient(hunting): picked mon.noname-b con 0x7feda413f710 addr v1:172.21.15.103:6789/0 2024-03-25T19:21:53.942 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.940+0000 7feda8aae640 10 monclient(hunting): picked mon.noname-c con 0x7feda41430c0 addr v1:172.21.15.83:6790/0 2024-03-25T19:21:53.942 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.940+0000 7feda8aae640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda4137f10 con 0x7feda40a98f0 2024-03-25T19:21:53.942 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.940+0000 7feda8aae640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda41dc070 con 0x7feda413f710 2024-03-25T19:21:53.942 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.940+0000 7feda8aae640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda4137100 con 0x7feda41430c0 2024-03-25T19:21:53.942 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.940+0000 7feda8aae640 10 monclient(hunting): _renew_subs 2024-03-25T19:21:53.942 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.940+0000 7feda8aae640 10 monclient(hunting): authenticate will time out at 6171.212180s 2024-03-25T19:21:53.949 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.949+0000 7feda1d9b640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7feda413f710 0x7feda41db250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:41020/0 (socket says 172.21.15.83:41020) 2024-03-25T19:21:53.949 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.949+0000 7feda1d9b640 1 -- 172.21.15.83:0/3418873641 learned_addr learned my addr 172.21.15.83:0/3418873641 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:21:53.950 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.950+0000 7fed9affd640 10 client.?.objecter ms_handle_connect 0x7feda413f710 2024-03-25T19:21:53.950 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.950+0000 7fed9affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:53.954 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.954+0000 7fed9affd640 1 -- 172.21.15.83:0/3418873641 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2434070031 0 0) 0x7feda41dc070 con 0x7feda413f710 2024-03-25T19:21:53.954 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.954+0000 7fed9affd640 10 monclient(hunting): _negotiate 2024-03-25T19:21:53.954 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.954+0000 7fed9affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:53.954 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.954+0000 7fed9affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:53.954 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.954+0000 7fed9affd640 10 monclient(hunting): my global_id is 4321 2024-03-25T19:21:53.955 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.954+0000 7fed9affd640 1 -- 172.21.15.83:0/3418873641 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed84004eb0 con 0x7feda413f710 2024-03-25T19:21:53.956 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.956+0000 7fed9affd640 10 client.?.objecter ms_handle_connect 0x7feda41430c0 2024-03-25T19:21:53.956 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.956+0000 7fed9affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:53.967 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.968+0000 7fed9affd640 10 client.?.objecter ms_handle_connect 0x7feda40a98f0 2024-03-25T19:21:53.967 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.968+0000 7fed9affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:53.975 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.975+0000 7fed9affd640 1 -- 172.21.15.83:0/3418873641 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3459280547 0 0) 0x7fed84004eb0 con 0x7feda413f710 2024-03-25T19:21:53.975 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.975+0000 7fed9affd640 1 -- 172.21.15.83:0/3418873641 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feda41dc070 con 0x7feda413f710 2024-03-25T19:21:53.975 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.975+0000 7fed9affd640 1 -- 172.21.15.83:0/3418873641 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fed8c001bc0 con 0x7feda413f710 2024-03-25T19:21:53.975 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.975+0000 7fed9affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:53.975 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.975+0000 7fed9affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:21:53.975 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.975+0000 7fed9affd640 10 monclient(hunting): dump: 2024-03-25T19:21:53.975 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:21:53.975 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:21:53.975 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:53.975 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:53.975 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:53.975 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:21:53.975 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:21:53.976 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:21:53.976 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:21:53.976 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:53.976 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.975+0000 7fed9affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:21:53.976 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.976+0000 7fed9affd640 1 -- 172.21.15.83:0/3418873641 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4087288668 0 0) 0x7feda4137100 con 0x7feda41430c0 2024-03-25T19:21:53.976 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.976+0000 7fed9affd640 10 monclient(hunting): _negotiate 2024-03-25T19:21:53.976 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.976+0000 7fed9affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:53.976 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.976+0000 7fed9affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:53.977 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.976+0000 7fed9affd640 10 monclient(hunting): my global_id is 4328 2024-03-25T19:21:53.977 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.976+0000 7fed9affd640 1 -- 172.21.15.83:0/3418873641 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed84004eb0 con 0x7feda41430c0 2024-03-25T19:21:53.984 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.984+0000 7fed9affd640 1 -- 172.21.15.83:0/3418873641 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4244405576 0 0) 0x7feda4137f10 con 0x7feda40a98f0 2024-03-25T19:21:53.984 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.984+0000 7fed9affd640 10 monclient(hunting): _negotiate 2024-03-25T19:21:53.984 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.984+0000 7fed9affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:53.984 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.984+0000 7fed9affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:53.985 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.984+0000 7fed9affd640 10 monclient(hunting): my global_id is 4317 2024-03-25T19:21:53.985 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.984+0000 7fed9affd640 1 -- 172.21.15.83:0/3418873641 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feda4137100 con 0x7feda40a98f0 2024-03-25T19:21:53.993 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.994+0000 7fed9affd640 1 -- 172.21.15.83:0/3418873641 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3224295088 0 0) 0x7fed84004eb0 con 0x7feda41430c0 2024-03-25T19:21:53.994 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.994+0000 7fed9affd640 1 -- 172.21.15.83:0/3418873641 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feda4137f10 con 0x7feda41430c0 2024-03-25T19:21:53.994 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.994+0000 7fed9affd640 1 -- 172.21.15.83:0/3418873641 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fed9c001a90 con 0x7feda41430c0 2024-03-25T19:21:53.994 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.994+0000 7fed9affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:53.994 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.994+0000 7fed9affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:21:53.994 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.994+0000 7fed9affd640 10 monclient(hunting): dump: 2024-03-25T19:21:53.994 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:21:53.994 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:21:53.994 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:53.994 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:53.994 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:53.994 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:21:53.994 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:21:53.994 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:21:53.994 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:21:53.994 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:53.996 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.996+0000 7fed9affd640 1 -- 172.21.15.83:0/3418873641 <== mon.1 v1:172.21.15.103:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4220456152 0 0) 0x7feda41dc070 con 0x7feda413f710 2024-03-25T19:21:53.996 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.996+0000 7fed9affd640 1 -- 172.21.15.83:0/3418873641 >> v1:172.21.15.83:6790/0 conn(0x7feda41430c0 legacy=0x7feda41db960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:53.996 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.996+0000 7fed9affd640 1 -- 172.21.15.83:0/3418873641 >> v1:172.21.15.83:6789/0 conn(0x7feda40a98f0 legacy=0x7feda4137800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:53.996 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.996+0000 7fed9affd640 10 monclient: _finish_hunting 0 2024-03-25T19:21:53.996 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.996+0000 7fed9affd640 1 monclient: found mon.b 2024-03-25T19:21:53.996 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.996+0000 7fed9affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:21:53.996 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.996+0000 7fed9affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:21:53.996 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.996+0000 7fed9affd640 1 -- 172.21.15.83:0/3418873641 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feda41373e0 con 0x7feda413f710 2024-03-25T19:21:53.997 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.996+0000 7fed9affd640 10 monclient: _finish_auth 0 2024-03-25T19:21:53.997 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.996+0000 7fed9affd640 10 monclient: _check_auth_tickets 2024-03-25T19:21:53.997 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.996+0000 7fed9affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:21:53.997 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.996+0000 7feda8aae640 5 monclient: authenticate success, global_id 4321 2024-03-25T19:21:53.997 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.996+0000 7feda8aae640 10 monclient: _renew_subs 2024-03-25T19:21:53.997 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.996+0000 7feda8aae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:21:53.997 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.996+0000 7feda8aae640 1 -- 172.21.15.83:0/3418873641 --> v1:172.21.15.103:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feda41358e0 con 0x7feda413f710 2024-03-25T19:21:53.997 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.996+0000 7feda8aae640 10 client.4321.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:21:53.997 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.996+0000 7feda8aae640 10 monclient: _renew_subs 2024-03-25T19:21:53.997 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.996+0000 7feda8aae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:21:53.997 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.996+0000 7feda8aae640 1 -- 172.21.15.83:0/3418873641 --> v1:172.21.15.103:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feda4136150 con 0x7feda413f710 2024-03-25T19:21:53.997 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.996+0000 7feda8aae640 1 librados: init done 2024-03-25T19:21:53.998 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.997+0000 7feda8aae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:53.998 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.997+0000 7feda8aae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:53.998 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.997+0000 7feda8aae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:21:53.998 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:53.997+0000 7feda8aae640 1 -- 172.21.15.83:0/3418873641 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feda40627b0 con 0x7feda413f710 2024-03-25T19:21:54.275 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:54.275+0000 7fed9affd640 1 -- 172.21.15.83:0/3418873641 <== mon.1 v1:172.21.15.103:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed8c0011f0 con 0x7feda413f710 2024-03-25T19:21:54.276 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:54.275+0000 7fed9affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:21:54.276 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:54.275+0000 7fed9affd640 1 -- 172.21.15.83:0/3418873641 <== mon.1 v1:172.21.15.103:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fed8c004c30 con 0x7feda413f710 2024-03-25T19:21:54.276 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:54.275+0000 7fed9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:54.276 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:54.275+0000 7fed9affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:21:54.276 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:54.275+0000 7fed9affd640 10 monclient: dump: 2024-03-25T19:21:54.276 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:21:54.276 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:21:54.276 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:54.276 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:21:54.276 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:54.276 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:21:54.276 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:21:54.276 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:21:54.277 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:21:54.277 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:21:54.277 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:54.275+0000 7feda0d99640 4 set_mon_vals no callback set 2024-03-25T19:21:54.277 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:54.276+0000 7fed9affd640 1 -- 172.21.15.83:0/3418873641 <== mon.1 v1:172.21.15.103:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7fed8c004e90 con 0x7feda413f710 2024-03-25T19:21:54.277 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:54.276+0000 7feda259c640 10 monclient: get_auth_request con 0x7fed8406df80 auth_method 0 2024-03-25T19:21:54.277 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:54.277+0000 7fed9affd640 10 client.4321.objecter ms_handle_connect 0x7fed8406df80 2024-03-25T19:21:54.281 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:54.281+0000 7fed9affd640 1 -- 172.21.15.83:0/3418873641 <== mon.1 v1:172.21.15.103:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7fed8c054a40 con 0x7feda413f710 2024-03-25T19:21:54.282 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:54.281+0000 7fed9affd640 10 client.4321.objecter ms_dispatch 0x7feda4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:21:54.282 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:54.281+0000 7fed9affd640 3 client.4321.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:21:54.282 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:54.281+0000 7fed9affd640 3 client.4321.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:21:54.282 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:54.281+0000 7fed9affd640 20 client.4321.objecter dump_active .. 0 homeless 2024-03-25T19:21:54.282 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:54.282+0000 7fed9affd640 1 -- 172.21.15.83:0/3418873641 <== mon.1 v1:172.21.15.103:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fed8c050380 con 0x7feda413f710 2024-03-25T19:21:54.282 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:54.282+0000 7fed9affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:54.282 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:54.282+0000 7fed9affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:21:54.542 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:54.542+0000 7feda8aae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:54.542 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:54.542+0000 7feda8aae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:54.542 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:54.542+0000 7feda8aae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:21:54.542 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:54.542+0000 7feda8aae640 1 -- 172.21.15.83:0/3418873641 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feda4135a70 con 0x7feda413f710 2024-03-25T19:21:54.576 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:54.576+0000 7fed9affd640 1 -- 172.21.15.83:0/3418873641 <== mon.1 v1:172.21.15.103:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7fed8c052480 con 0x7feda413f710 2024-03-25T19:21:54.576 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:54.576+0000 7fed9affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:54.576 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:54.576+0000 7fed9affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:21:54.576 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:21:54.576 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:21:54.579 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:54.579+0000 7feda8aae640 10 librados: watch_flush enter 2024-03-25T19:21:54.579 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:54.579+0000 7feda8aae640 10 librados: watch_flush exit 2024-03-25T19:21:54.579 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:54.579+0000 7feda8aae640 20 client.4321.objecter shutdown clearing up homeless session... 2024-03-25T19:21:54.579 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:54.579+0000 7feda8aae640 10 client.4321.objecter successfully canceled tick 2024-03-25T19:21:54.579 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:54.579+0000 7feda8aae640 1 -- 172.21.15.83:0/3418873641 >> v1:172.21.15.103:6800/345072327 conn(0x7fed8406df80 legacy=0x7fed84070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:54.579 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:54.579+0000 7feda8aae640 10 monclient: shutdown 2024-03-25T19:21:54.579 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:54.579+0000 7feda8aae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:21:54.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:54.579+0000 7feda8aae640 1 -- 172.21.15.83:0/3418873641 >> v1:172.21.15.103:6789/0 conn(0x7feda413f710 legacy=0x7feda41db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:54.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:54.579+0000 7feda8aae640 1 -- 172.21.15.83:0/3418873641 shutdown_connections 2024-03-25T19:21:54.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:54.579+0000 7feda8aae640 1 -- 172.21.15.83:0/3418873641 >> 172.21.15.83:0/3418873641 conn(0x7feda40b7ea0 msgr2=0x7feda414ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:21:54.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:54.579+0000 7feda8aae640 1 -- 172.21.15.83:0/3418873641 shutdown_connections 2024-03-25T19:21:54.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:54.579+0000 7feda8aae640 1 -- 172.21.15.83:0/3418873641 wait complete. 2024-03-25T19:21:54.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:21:54.579+0000 7feda8aae640 1 librados: shutdown 2024-03-25T19:21:54.591 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:22:01.398 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:22:01.514 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.514+0000 7f209174b640 10 monclient: get_monmap_and_config 2024-03-25T19:22:01.514 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.514+0000 7f209174b640 10 monclient: build_initial_monmap 2024-03-25T19:22:01.514 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.514+0000 7f209174b640 10 monclient: monmap: 2024-03-25T19:22:01.514 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:22:01.514 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:22:01.514 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:22:01.515557+0000 2024-03-25T19:22:01.514 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:22:01.515557+0000 2024-03-25T19:22:01.514 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:01.514 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:22:01.515 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:22:01.515 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:22:01.515 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:22:01.515 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:01.516 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.515+0000 7f209174b640 1 Processor -- start 2024-03-25T19:22:01.516 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.515+0000 7f209174b640 1 -- start start 2024-03-25T19:22:01.516 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.515+0000 7f209174b640 10 monclient: init 2024-03-25T19:22:01.516 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.516+0000 7f209174b640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:01.516 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.516+0000 7f209174b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:22:01.516 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.516+0000 7f209174b640 10 monclient(hunting): picked mon.noname-c con 0x7f208c0b4d60 addr v1:172.21.15.103:6789/0 2024-03-25T19:22:01.516 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.516+0000 7f209174b640 10 monclient(hunting): picked mon.noname-b con 0x7f208c0b1730 addr v1:172.21.15.83:6790/0 2024-03-25T19:22:01.516 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.516+0000 7f209174b640 10 monclient(hunting): picked mon.noname-a con 0x7f208c14be90 addr v1:172.21.15.83:6789/0 2024-03-25T19:22:01.516 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.516+0000 7f209174b640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f208c0af940 con 0x7f208c14be90 2024-03-25T19:22:01.516 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.516+0000 7f209174b640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f208c0b0a70 con 0x7f208c0b4d60 2024-03-25T19:22:01.517 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.516+0000 7f209174b640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f208c0b0d50 con 0x7f208c0b1730 2024-03-25T19:22:01.517 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.516+0000 7f209174b640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:01.517 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.516+0000 7f209174b640 10 monclient(hunting): authenticate will time out at 6178.788100s 2024-03-25T19:22:01.523 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.524+0000 7f208b7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f208c14be90 0x7f208c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:37142/0 (socket says 172.21.15.83:37142) 2024-03-25T19:22:01.524 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.524+0000 7f208b7fe640 1 -- 172.21.15.83:0/2293561477 learned_addr learned my addr 172.21.15.83:0/2293561477 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:22:01.532 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.532+0000 7f208a7fc640 1 -- 172.21.15.83:0/2293561477 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3539445896 0 0) 0x7f208c0af940 con 0x7f208c14be90 2024-03-25T19:22:01.532 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.532+0000 7f208a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:22:01.532 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.532+0000 7f208a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:01.532 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.532+0000 7f208a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:01.532 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.532+0000 7f208a7fc640 10 monclient(hunting): my global_id is 4320 2024-03-25T19:22:01.532 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.532+0000 7f208a7fc640 1 -- 172.21.15.83:0/2293561477 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2078003520 con 0x7f208c14be90 2024-03-25T19:22:01.543 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.543+0000 7f208a7fc640 1 -- 172.21.15.83:0/2293561477 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1506785281 0 0) 0x7f208c0b0d50 con 0x7f208c0b1730 2024-03-25T19:22:01.543 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.543+0000 7f208a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:22:01.543 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.543+0000 7f208a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:01.543 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.543+0000 7f208a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:01.543 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.543+0000 7f208a7fc640 10 monclient(hunting): my global_id is 4331 2024-03-25T19:22:01.543 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.544+0000 7f208a7fc640 1 -- 172.21.15.83:0/2293561477 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f208c0af940 con 0x7f208c0b1730 2024-03-25T19:22:01.545 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.545+0000 7f208a7fc640 1 -- 172.21.15.83:0/2293561477 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 309240900 0 0) 0x7f2078003520 con 0x7f208c14be90 2024-03-25T19:22:01.545 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.545+0000 7f208a7fc640 1 -- 172.21.15.83:0/2293561477 >> v1:172.21.15.83:6790/0 conn(0x7f208c0b1730 legacy=0x7f208c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:01.545 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.545+0000 7f208a7fc640 1 -- 172.21.15.83:0/2293561477 >> v1:172.21.15.103:6789/0 conn(0x7f208c0b4d60 legacy=0x7f208c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:01.545 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.545+0000 7f208a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:22:01.545 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.545+0000 7f208a7fc640 1 monclient: found mon.noname-a 2024-03-25T19:22:01.545 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.545+0000 7f208a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:01.545 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.545+0000 7f208a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.83:6789/0 2024-03-25T19:22:01.545 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.545+0000 7f208a7fc640 1 -- 172.21.15.83:0/2293561477 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f208c0b1030 con 0x7f208c14be90 2024-03-25T19:22:01.545 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.545+0000 7f208a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:22:01.545 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.545+0000 7f208a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:22:01.545 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.545+0000 7f208a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:01.545 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.545+0000 7f208a7fc640 1 -- 172.21.15.83:0/2293561477 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f2084001a10 con 0x7f208c14be90 2024-03-25T19:22:01.546 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.545+0000 7f208a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:01.546 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.545+0000 7f208a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:22:01.546 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.545+0000 7f208a7fc640 10 monclient: dump: 2024-03-25T19:22:01.546 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:22:01.546 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:22:01.546 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:01.546 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:01.546 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:01.546 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:22:01.546 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:22:01.546 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:22:01.546 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:22:01.546 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:01.546 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.545+0000 7f209174b640 5 monclient: authenticate success, global_id 4320 2024-03-25T19:22:01.547 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.545+0000 7f209174b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:01.553 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.553+0000 7f208a7fc640 1 -- 172.21.15.83:0/2293561477 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2084001b70 con 0x7f208c14be90 2024-03-25T19:22:01.553 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.553+0000 7f208a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:01.553 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.553+0000 7f208a7fc640 1 -- 172.21.15.83:0/2293561477 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f2084001f80 con 0x7f208c14be90 2024-03-25T19:22:01.553 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.553+0000 7f208a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:01.553 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.553+0000 7f208a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:22:01.553 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.553+0000 7f208a7fc640 10 monclient: dump: 2024-03-25T19:22:01.553 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:22:01.553 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:22:01.553 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:01.553 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:01.553 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:01.553 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:22:01.553 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:22:01.554 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:22:01.554 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:22:01.554 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:01.554 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.553+0000 7f209174b640 10 monclient: get_monmap_and_config success 2024-03-25T19:22:01.554 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.553+0000 7f209174b640 4 set_mon_vals no callback set 2024-03-25T19:22:01.556 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.554+0000 7f209174b640 10 monclient: shutdown 2024-03-25T19:22:01.556 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.554+0000 7f209174b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:01.556 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.554+0000 7f209174b640 1 -- 172.21.15.83:0/2293561477 >> v1:172.21.15.83:6789/0 conn(0x7f208c14be90 legacy=0x7f208c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:01.556 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.554+0000 7f209174b640 1 -- 172.21.15.83:0/2293561477 shutdown_connections 2024-03-25T19:22:01.556 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.554+0000 7f209174b640 1 -- 172.21.15.83:0/2293561477 >> 172.21.15.83:0/2293561477 conn(0x7f208c0617a0 msgr2=0x7f208c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:01.556 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.554+0000 7f209174b640 1 -- 172.21.15.83:0/2293561477 shutdown_connections 2024-03-25T19:22:01.556 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.554+0000 7f209174b640 1 -- 172.21.15.83:0/2293561477 wait complete. 2024-03-25T19:22:01.556 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.555+0000 7f209174b640 10 monclient: build_initial_monmap 2024-03-25T19:22:01.556 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.555+0000 7f209174b640 10 monclient: monmap: 2024-03-25T19:22:01.556 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:22:01.556 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:22:01.556 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:22:01.557 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:22:01.557 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:01.557 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:22:01.557 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:22:01.557 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:22:01.557 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:22:01.557 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:01.557 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.555+0000 7f209174b640 1 librados: starting msgr at 2024-03-25T19:22:01.557 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.555+0000 7f209174b640 1 librados: starting objecter 2024-03-25T19:22:01.557 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.555+0000 7f209174b640 1 Processor -- start 2024-03-25T19:22:01.557 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.555+0000 7f209174b640 1 -- start start 2024-03-25T19:22:01.557 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.555+0000 7f209174b640 1 librados: setting wanted keys 2024-03-25T19:22:01.557 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.555+0000 7f209174b640 1 librados: calling monclient init 2024-03-25T19:22:01.557 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.555+0000 7f209174b640 10 monclient: init 2024-03-25T19:22:01.557 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.556+0000 7f209174b640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:01.557 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.556+0000 7f209174b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:22:01.558 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.556+0000 7f209174b640 10 monclient(hunting): picked mon.noname-c con 0x7f208c0b1730 addr v1:172.21.15.83:6790/0 2024-03-25T19:22:01.558 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.556+0000 7f209174b640 10 monclient(hunting): picked mon.noname-a con 0x7f208c0b4d60 addr v1:172.21.15.83:6789/0 2024-03-25T19:22:01.558 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.556+0000 7f209174b640 10 monclient(hunting): picked mon.noname-b con 0x7f208c14be90 addr v1:172.21.15.103:6789/0 2024-03-25T19:22:01.558 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.556+0000 7f209174b640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f208c138100 con 0x7f208c0b4d60 2024-03-25T19:22:01.558 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.556+0000 7f209174b640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f208c1383e0 con 0x7f208c14be90 2024-03-25T19:22:01.558 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.556+0000 7f209174b640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f208c136db0 con 0x7f208c0b1730 2024-03-25T19:22:01.558 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.556+0000 7f209174b640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:01.558 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.556+0000 7f209174b640 10 monclient(hunting): authenticate will time out at 6178.828301s 2024-03-25T19:22:01.561 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.561+0000 7f208b7fe640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f208c14be90 0x7f208c1e3540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:55964/0 (socket says 172.21.15.83:55964) 2024-03-25T19:22:01.561 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.561+0000 7f208b7fe640 1 -- 172.21.15.83:0/3201989923 learned_addr learned my addr 172.21.15.83:0/3201989923 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:22:01.562 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.562+0000 7f20837fe640 10 client.?.objecter ms_handle_connect 0x7f208c14be90 2024-03-25T19:22:01.562 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.562+0000 7f20837fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:01.569 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.570+0000 7f20837fe640 1 -- 172.21.15.83:0/3201989923 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1020156993 0 0) 0x7f208c1383e0 con 0x7f208c14be90 2024-03-25T19:22:01.570 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.570+0000 7f20837fe640 10 monclient(hunting): _negotiate 2024-03-25T19:22:01.570 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.570+0000 7f20837fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:01.570 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.570+0000 7f20837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:01.570 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.570+0000 7f20837fe640 10 monclient(hunting): my global_id is 4327 2024-03-25T19:22:01.570 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.570+0000 7f20837fe640 1 -- 172.21.15.83:0/3201989923 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f206c004eb0 con 0x7f208c14be90 2024-03-25T19:22:01.573 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.573+0000 7f20837fe640 10 client.?.objecter ms_handle_connect 0x7f208c0b1730 2024-03-25T19:22:01.573 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.573+0000 7f20837fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:01.579 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.579+0000 7f20837fe640 10 client.?.objecter ms_handle_connect 0x7f208c0b4d60 2024-03-25T19:22:01.579 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.579+0000 7f20837fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:01.585 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.585+0000 7f20837fe640 1 -- 172.21.15.83:0/3201989923 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3413530715 0 0) 0x7f206c004eb0 con 0x7f208c14be90 2024-03-25T19:22:01.585 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.585+0000 7f20837fe640 1 -- 172.21.15.83:0/3201989923 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f208c1383e0 con 0x7f208c14be90 2024-03-25T19:22:01.585 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.585+0000 7f20837fe640 1 -- 172.21.15.83:0/3201989923 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f2084001ec0 con 0x7f208c14be90 2024-03-25T19:22:01.585 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.585+0000 7f20837fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:01.585 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.585+0000 7f20837fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:22:01.586 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.585+0000 7f20837fe640 10 monclient(hunting): dump: 2024-03-25T19:22:01.586 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:22:01.586 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:22:01.586 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:01.586 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:01.586 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:01.586 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:22:01.586 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:22:01.586 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:22:01.586 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:22:01.586 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:01.586 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.585+0000 7f20837fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:22:01.588 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.588+0000 7f20837fe640 1 -- 172.21.15.83:0/3201989923 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2944644506 0 0) 0x7f208c138100 con 0x7f208c0b4d60 2024-03-25T19:22:01.588 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.588+0000 7f20837fe640 10 monclient(hunting): _negotiate 2024-03-25T19:22:01.588 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.588+0000 7f20837fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:01.588 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.588+0000 7f20837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:01.588 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.588+0000 7f20837fe640 10 monclient(hunting): my global_id is 4323 2024-03-25T19:22:01.588 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.588+0000 7f20837fe640 1 -- 172.21.15.83:0/3201989923 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f206c004eb0 con 0x7f208c0b4d60 2024-03-25T19:22:01.594 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.594+0000 7f20837fe640 1 -- 172.21.15.83:0/3201989923 <== mon.1 v1:172.21.15.103:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2152095169 0 0) 0x7f208c1383e0 con 0x7f208c14be90 2024-03-25T19:22:01.594 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.594+0000 7f20837fe640 1 -- 172.21.15.83:0/3201989923 >> v1:172.21.15.83:6790/0 conn(0x7f208c0b1730 legacy=0x7f208c1427d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:01.594 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.594+0000 7f20837fe640 1 -- 172.21.15.83:0/3201989923 >> v1:172.21.15.83:6789/0 conn(0x7f208c0b4d60 legacy=0x7f208c143900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:01.594 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.594+0000 7f20837fe640 10 monclient: _finish_hunting 0 2024-03-25T19:22:01.594 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.594+0000 7f20837fe640 1 monclient: found mon.b 2024-03-25T19:22:01.594 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.594+0000 7f20837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:01.594 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.594+0000 7f20837fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:22:01.594 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.594+0000 7f20837fe640 1 -- 172.21.15.83:0/3201989923 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f208c135ff0 con 0x7f208c14be90 2024-03-25T19:22:01.595 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.594+0000 7f20837fe640 10 monclient: _finish_auth 0 2024-03-25T19:22:01.595 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.594+0000 7f20837fe640 10 monclient: _check_auth_tickets 2024-03-25T19:22:01.595 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.594+0000 7f20837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:01.595 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.594+0000 7f209174b640 5 monclient: authenticate success, global_id 4327 2024-03-25T19:22:01.595 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.594+0000 7f209174b640 10 monclient: _renew_subs 2024-03-25T19:22:01.595 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.594+0000 7f209174b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:22:01.595 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.594+0000 7f209174b640 1 -- 172.21.15.83:0/3201989923 --> v1:172.21.15.103:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f208c136240 con 0x7f208c14be90 2024-03-25T19:22:01.595 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.594+0000 7f209174b640 10 client.4327.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:22:01.595 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.594+0000 7f209174b640 10 monclient: _renew_subs 2024-03-25T19:22:01.595 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.594+0000 7f209174b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:22:01.595 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.594+0000 7f209174b640 1 -- 172.21.15.83:0/3201989923 --> v1:172.21.15.103:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f208c136900 con 0x7f208c14be90 2024-03-25T19:22:01.595 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.594+0000 7f209174b640 1 librados: init done 2024-03-25T19:22:01.596 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.595+0000 7f209174b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:01.596 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.595+0000 7f209174b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:01.596 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.595+0000 7f209174b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:22:01.596 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.595+0000 7f209174b640 1 -- 172.21.15.83:0/3201989923 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f208c136f40 con 0x7f208c14be90 2024-03-25T19:22:01.887 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.887+0000 7f20837fe640 1 -- 172.21.15.83:0/3201989923 <== mon.1 v1:172.21.15.103:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2084004e10 con 0x7f208c14be90 2024-03-25T19:22:01.887 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.887+0000 7f20837fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:01.887 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.887+0000 7f20837fe640 1 -- 172.21.15.83:0/3201989923 <== mon.1 v1:172.21.15.103:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f2084005120 con 0x7f208c14be90 2024-03-25T19:22:01.887 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.887+0000 7f20837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:01.887 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.887+0000 7f20837fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:22:01.887 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.887+0000 7f20837fe640 10 monclient: dump: 2024-03-25T19:22:01.887 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:22:01.887 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:22:01.887 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:01.888 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:01.888 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:01.888 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:22:01.888 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:22:01.888 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:22:01.888 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:22:01.888 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:01.888 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.887+0000 7f2089ffb640 4 set_mon_vals no callback set 2024-03-25T19:22:01.889 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.888+0000 7f20837fe640 1 -- 172.21.15.83:0/3201989923 <== mon.1 v1:172.21.15.103:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7f208401b0f0 con 0x7f208c14be90 2024-03-25T19:22:01.889 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.889+0000 7f208affd640 10 monclient: get_auth_request con 0x7f206c06dd90 auth_method 0 2024-03-25T19:22:01.889 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.889+0000 7f20837fe640 10 client.4327.objecter ms_handle_connect 0x7f206c06dd90 2024-03-25T19:22:01.893 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.893+0000 7f20837fe640 1 -- 172.21.15.83:0/3201989923 <== mon.1 v1:172.21.15.103:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7f2084054b20 con 0x7f208c14be90 2024-03-25T19:22:01.893 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.893+0000 7f20837fe640 10 client.4327.objecter ms_dispatch 0x7f208c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:22:01.893 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.893+0000 7f20837fe640 3 client.4327.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:22:01.893 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.893+0000 7f20837fe640 3 client.4327.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:22:01.893 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.893+0000 7f20837fe640 20 client.4327.objecter dump_active .. 0 homeless 2024-03-25T19:22:01.894 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.894+0000 7f20837fe640 1 -- 172.21.15.83:0/3201989923 <== mon.1 v1:172.21.15.103:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2084050630 con 0x7f208c14be90 2024-03-25T19:22:01.894 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.894+0000 7f20837fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:01.894 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:01.894+0000 7f20837fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:22:02.141 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:02.141+0000 7f209174b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:02.141 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:02.141+0000 7f209174b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:02.141 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:02.141+0000 7f209174b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:22:02.142 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:02.141+0000 7f209174b640 1 -- 172.21.15.83:0/3201989923 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f208c0b2270 con 0x7f208c14be90 2024-03-25T19:22:02.168 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:02.168+0000 7f20837fe640 1 -- 172.21.15.83:0/3201989923 <== mon.1 v1:172.21.15.103:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7f2084052560 con 0x7f208c14be90 2024-03-25T19:22:02.168 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:02.168+0000 7f20837fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:02.168 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:02.168+0000 7f20837fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:22:02.168 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:22:02.169 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:22:02.171 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:02.170+0000 7f209174b640 10 librados: watch_flush enter 2024-03-25T19:22:02.171 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:02.171+0000 7f209174b640 10 librados: watch_flush exit 2024-03-25T19:22:02.171 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:02.171+0000 7f209174b640 20 client.4327.objecter shutdown clearing up homeless session... 2024-03-25T19:22:02.171 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:02.171+0000 7f209174b640 10 client.4327.objecter successfully canceled tick 2024-03-25T19:22:02.171 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:02.171+0000 7f209174b640 1 -- 172.21.15.83:0/3201989923 >> v1:172.21.15.103:6800/345072327 conn(0x7f206c06dd90 legacy=0x7f206c070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:02.171 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:02.171+0000 7f209174b640 10 monclient: shutdown 2024-03-25T19:22:02.171 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:02.171+0000 7f209174b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:02.171 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:02.171+0000 7f209174b640 1 -- 172.21.15.83:0/3201989923 >> v1:172.21.15.103:6789/0 conn(0x7f208c14be90 legacy=0x7f208c1e3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:02.171 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:02.171+0000 7f209174b640 1 -- 172.21.15.83:0/3201989923 shutdown_connections 2024-03-25T19:22:02.172 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:02.171+0000 7f209174b640 1 -- 172.21.15.83:0/3201989923 >> 172.21.15.83:0/3201989923 conn(0x7f208c0617a0 msgr2=0x7f208c13c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:02.172 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:02.171+0000 7f209174b640 1 -- 172.21.15.83:0/3201989923 shutdown_connections 2024-03-25T19:22:02.172 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:02.171+0000 7f209174b640 1 -- 172.21.15.83:0/3201989923 wait complete. 2024-03-25T19:22:02.172 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:02.171+0000 7f209174b640 1 librados: shutdown 2024-03-25T19:22:02.183 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:22:02.487 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:02:44.414 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:02.493 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:02:44.421 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:02.527 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:02:44.454 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:02.554 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:02:44.465 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:02.558 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:02:44.469 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:02.567 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:02:44.494 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:02.574 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:02:44.501 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:02.591 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:02:44.501 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:02.603 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:02:44.531 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:08.990 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:22:09.101 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.101+0000 7fea5d132640 10 monclient: get_monmap_and_config 2024-03-25T19:22:09.101 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.101+0000 7fea5d132640 10 monclient: build_initial_monmap 2024-03-25T19:22:09.101 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.101+0000 7fea5d132640 10 monclient: monmap: 2024-03-25T19:22:09.101 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:22:09.101 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:22:09.101 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:22:09.102345+0000 2024-03-25T19:22:09.101 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:22:09.102345+0000 2024-03-25T19:22:09.101 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:09.101 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:22:09.102 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:22:09.102 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:22:09.102 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:22:09.102 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:09.103 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.102+0000 7fea5d132640 1 Processor -- start 2024-03-25T19:22:09.103 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.102+0000 7fea5d132640 1 -- start start 2024-03-25T19:22:09.103 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.102+0000 7fea5d132640 10 monclient: init 2024-03-25T19:22:09.103 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.103+0000 7fea5d132640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:09.103 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.103+0000 7fea5d132640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:22:09.103 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.103+0000 7fea5d132640 10 monclient(hunting): picked mon.noname-a con 0x7fea580a98f0 addr v1:172.21.15.83:6789/0 2024-03-25T19:22:09.104 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.103+0000 7fea5d132640 10 monclient(hunting): picked mon.noname-b con 0x7fea5813f710 addr v1:172.21.15.83:6790/0 2024-03-25T19:22:09.104 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.103+0000 7fea5d132640 10 monclient(hunting): picked mon.noname-c con 0x7fea581430c0 addr v1:172.21.15.103:6789/0 2024-03-25T19:22:09.104 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.103+0000 7fea5d132640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea58142520 con 0x7fea580a98f0 2024-03-25T19:22:09.104 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.103+0000 7fea5d132640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea580b28a0 con 0x7fea581430c0 2024-03-25T19:22:09.104 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.103+0000 7fea5d132640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea580b2a30 con 0x7fea5813f710 2024-03-25T19:22:09.104 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.103+0000 7fea5d132640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:09.104 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.103+0000 7fea5d132640 10 monclient(hunting): authenticate will time out at 6186.375017s 2024-03-25T19:22:09.104 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.103+0000 7fea5759e640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fea581430c0 0x7fea5814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:59620/0 (socket says 172.21.15.83:59620) 2024-03-25T19:22:09.104 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.103+0000 7fea5759e640 1 -- 172.21.15.83:0/802441270 learned_addr learned my addr 172.21.15.83:0/802441270 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:22:09.179 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.179+0000 7fea5659c640 1 -- 172.21.15.83:0/802441270 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1961623753 0 0) 0x7fea580b28a0 con 0x7fea581430c0 2024-03-25T19:22:09.179 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.179+0000 7fea5659c640 10 monclient(hunting): _negotiate 2024-03-25T19:22:09.179 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.179+0000 7fea5659c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:09.179 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.179+0000 7fea5659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:09.179 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.179+0000 7fea5659c640 10 monclient(hunting): my global_id is 4330 2024-03-25T19:22:09.179 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.179+0000 7fea5659c640 1 -- 172.21.15.83:0/802441270 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea44003520 con 0x7fea581430c0 2024-03-25T19:22:09.191 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.191+0000 7fea5659c640 1 -- 172.21.15.83:0/802441270 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 755227066 0 0) 0x7fea44003520 con 0x7fea581430c0 2024-03-25T19:22:09.191 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.191+0000 7fea5659c640 1 -- 172.21.15.83:0/802441270 >> v1:172.21.15.83:6790/0 conn(0x7fea5813f710 legacy=0x7fea581429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:09.191 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.191+0000 7fea5659c640 1 -- 172.21.15.83:0/802441270 >> v1:172.21.15.83:6789/0 conn(0x7fea580a98f0 legacy=0x7fea580a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:09.191 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.191+0000 7fea5659c640 10 monclient: _finish_hunting 0 2024-03-25T19:22:09.191 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.191+0000 7fea5659c640 1 monclient: found mon.noname-c 2024-03-25T19:22:09.191 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.191+0000 7fea5659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:09.191 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.191+0000 7fea5659c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.103:6789/0 2024-03-25T19:22:09.191 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.191+0000 7fea5659c640 1 -- 172.21.15.83:0/802441270 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea580b44c0 con 0x7fea581430c0 2024-03-25T19:22:09.191 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.191+0000 7fea5659c640 10 monclient: _finish_auth 0 2024-03-25T19:22:09.191 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.191+0000 7fea5659c640 10 monclient: _check_auth_tickets 2024-03-25T19:22:09.191 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.191+0000 7fea5659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:09.191 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.191+0000 7fea5659c640 1 -- 172.21.15.83:0/802441270 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fea50001a10 con 0x7fea581430c0 2024-03-25T19:22:09.192 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.191+0000 7fea5d132640 5 monclient: authenticate success, global_id 4330 2024-03-25T19:22:09.192 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.191+0000 7fea5d132640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:09.192 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.191+0000 7fea5659c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:09.192 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.191+0000 7fea5659c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:22:09.192 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.191+0000 7fea5659c640 10 monclient: dump: 2024-03-25T19:22:09.192 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:22:09.192 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:22:09.192 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:09.192 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:09.192 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:09.192 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:22:09.192 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:22:09.192 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:22:09.192 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:22:09.192 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:09.192 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.191+0000 7fea5d132640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:09.206 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.206+0000 7fea5659c640 1 -- 172.21.15.83:0/802441270 <== mon.1 v1:172.21.15.103:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea50001b70 con 0x7fea581430c0 2024-03-25T19:22:09.206 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.206+0000 7fea5659c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:09.206 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.206+0000 7fea5659c640 1 -- 172.21.15.83:0/802441270 <== mon.1 v1:172.21.15.103:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fea50001f80 con 0x7fea581430c0 2024-03-25T19:22:09.206 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.206+0000 7fea5659c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:09.206 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.206+0000 7fea5659c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:22:09.206 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.206+0000 7fea5659c640 10 monclient: dump: 2024-03-25T19:22:09.206 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:22:09.206 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:22:09.206 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:09.206 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:09.207 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:09.207 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:22:09.207 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:22:09.207 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:22:09.207 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:22:09.207 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:09.207 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.206+0000 7fea5d132640 10 monclient: get_monmap_and_config success 2024-03-25T19:22:09.207 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.206+0000 7fea5d132640 4 set_mon_vals no callback set 2024-03-25T19:22:09.208 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.207+0000 7fea5d132640 10 monclient: shutdown 2024-03-25T19:22:09.208 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.207+0000 7fea5d132640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:09.208 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.207+0000 7fea5d132640 1 -- 172.21.15.83:0/802441270 >> v1:172.21.15.103:6789/0 conn(0x7fea581430c0 legacy=0x7fea5814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:09.208 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.207+0000 7fea5d132640 1 -- 172.21.15.83:0/802441270 shutdown_connections 2024-03-25T19:22:09.208 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.207+0000 7fea5d132640 1 -- 172.21.15.83:0/802441270 >> 172.21.15.83:0/802441270 conn(0x7fea580b7ea0 msgr2=0x7fea580b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:09.208 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.207+0000 7fea5d132640 1 -- 172.21.15.83:0/802441270 shutdown_connections 2024-03-25T19:22:09.208 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.207+0000 7fea5d132640 1 -- 172.21.15.83:0/802441270 wait complete. 2024-03-25T19:22:09.208 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.208+0000 7fea5d132640 10 monclient: build_initial_monmap 2024-03-25T19:22:09.208 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.208+0000 7fea5d132640 10 monclient: monmap: 2024-03-25T19:22:09.208 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:22:09.209 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:22:09.209 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:22:09.209 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:22:09.209 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:09.209 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:22:09.209 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:22:09.209 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:22:09.209 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:22:09.209 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:09.210 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.209+0000 7fea5d132640 1 librados: starting msgr at 2024-03-25T19:22:09.211 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.209+0000 7fea5d132640 1 librados: starting objecter 2024-03-25T19:22:09.211 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.209+0000 7fea5d132640 1 Processor -- start 2024-03-25T19:22:09.211 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.209+0000 7fea5d132640 1 -- start start 2024-03-25T19:22:09.211 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.209+0000 7fea5d132640 1 librados: setting wanted keys 2024-03-25T19:22:09.211 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.209+0000 7fea5d132640 1 librados: calling monclient init 2024-03-25T19:22:09.211 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.209+0000 7fea5d132640 10 monclient: init 2024-03-25T19:22:09.211 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.210+0000 7fea5d132640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:09.211 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.210+0000 7fea5d132640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:22:09.211 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.210+0000 7fea5d132640 10 monclient(hunting): picked mon.noname-b con 0x7fea580a98f0 addr v1:172.21.15.103:6789/0 2024-03-25T19:22:09.211 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.210+0000 7fea5d132640 10 monclient(hunting): picked mon.noname-c con 0x7fea5813f710 addr v1:172.21.15.83:6790/0 2024-03-25T19:22:09.211 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.210+0000 7fea5d132640 10 monclient(hunting): picked mon.noname-a con 0x7fea581430c0 addr v1:172.21.15.83:6789/0 2024-03-25T19:22:09.211 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.210+0000 7fea5d132640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea58136ad0 con 0x7fea581430c0 2024-03-25T19:22:09.211 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.210+0000 7fea5d132640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea580b7520 con 0x7fea580a98f0 2024-03-25T19:22:09.211 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.210+0000 7fea5d132640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea58135810 con 0x7fea5813f710 2024-03-25T19:22:09.211 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.210+0000 7fea5d132640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:09.212 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.210+0000 7fea5d132640 10 monclient(hunting): authenticate will time out at 6186.482474s 2024-03-25T19:22:09.216 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.217+0000 7fea5759e640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fea581430c0 0x7fea58130f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:45882/0 (socket says 172.21.15.83:45882) 2024-03-25T19:22:09.216 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.217+0000 7fea5759e640 1 -- 172.21.15.83:0/3302568104 learned_addr learned my addr 172.21.15.83:0/3302568104 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:22:09.222 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.223+0000 7fea4f7fe640 10 client.?.objecter ms_handle_connect 0x7fea580a98f0 2024-03-25T19:22:09.222 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.223+0000 7fea4f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:09.224 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.224+0000 7fea4f7fe640 10 client.?.objecter ms_handle_connect 0x7fea581430c0 2024-03-25T19:22:09.224 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.224+0000 7fea4f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:09.235 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.236+0000 7fea4f7fe640 1 -- 172.21.15.83:0/3302568104 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3572988483 0 0) 0x7fea58136ad0 con 0x7fea581430c0 2024-03-25T19:22:09.235 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.236+0000 7fea4f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:22:09.235 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.236+0000 7fea4f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:09.235 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.236+0000 7fea4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:09.235 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.236+0000 7fea4f7fe640 10 monclient(hunting): my global_id is 4329 2024-03-25T19:22:09.236 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.236+0000 7fea4f7fe640 1 -- 172.21.15.83:0/3302568104 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea38004eb0 con 0x7fea581430c0 2024-03-25T19:22:09.238 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.238+0000 7fea4f7fe640 1 -- 172.21.15.83:0/3302568104 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3212140622 0 0) 0x7fea580b7520 con 0x7fea580a98f0 2024-03-25T19:22:09.238 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.238+0000 7fea4f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:22:09.238 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.238+0000 7fea4f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:09.238 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.238+0000 7fea4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:09.238 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.238+0000 7fea4f7fe640 10 monclient(hunting): my global_id is 4333 2024-03-25T19:22:09.238 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.238+0000 7fea4f7fe640 1 -- 172.21.15.83:0/3302568104 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea58136ad0 con 0x7fea580a98f0 2024-03-25T19:22:09.253 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.254+0000 7fea4f7fe640 10 client.?.objecter ms_handle_connect 0x7fea5813f710 2024-03-25T19:22:09.253 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.254+0000 7fea4f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:09.262 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.262+0000 7fea4f7fe640 1 -- 172.21.15.83:0/3302568104 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1923135615 0 0) 0x7fea38004eb0 con 0x7fea581430c0 2024-03-25T19:22:09.262 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.262+0000 7fea4f7fe640 1 -- 172.21.15.83:0/3302568104 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea580b7520 con 0x7fea581430c0 2024-03-25T19:22:09.262 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.262+0000 7fea4f7fe640 1 -- 172.21.15.83:0/3302568104 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fea50001f40 con 0x7fea581430c0 2024-03-25T19:22:09.262 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.262+0000 7fea4f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:09.262 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.262+0000 7fea4f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:22:09.262 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.262+0000 7fea4f7fe640 10 monclient(hunting): dump: 2024-03-25T19:22:09.262 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:22:09.262 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:22:09.262 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:09.262 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:09.262 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:09.262 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:22:09.262 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:22:09.263 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:22:09.263 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:22:09.263 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:09.263 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.262+0000 7fea4f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:22:09.265 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.265+0000 7fea4f7fe640 1 -- 172.21.15.83:0/3302568104 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3455709414 0 0) 0x7fea58136ad0 con 0x7fea580a98f0 2024-03-25T19:22:09.265 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.265+0000 7fea4f7fe640 1 -- 172.21.15.83:0/3302568104 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea38004eb0 con 0x7fea580a98f0 2024-03-25T19:22:09.265 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.265+0000 7fea4f7fe640 1 -- 172.21.15.83:0/3302568104 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fea48001930 con 0x7fea580a98f0 2024-03-25T19:22:09.265 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.265+0000 7fea4f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:09.265 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.265+0000 7fea4f7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:22:09.265 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.265+0000 7fea4f7fe640 10 monclient(hunting): dump: 2024-03-25T19:22:09.265 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:22:09.265 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:22:09.266 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:09.266 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:09.266 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:09.266 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:22:09.266 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:22:09.266 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:22:09.266 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:22:09.266 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:09.267 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.267+0000 7fea4f7fe640 1 -- 172.21.15.83:0/3302568104 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2333925177 0 0) 0x7fea58135810 con 0x7fea5813f710 2024-03-25T19:22:09.267 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.267+0000 7fea4f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:22:09.267 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.267+0000 7fea4f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:09.267 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.267+0000 7fea4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:09.267 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.267+0000 7fea4f7fe640 10 monclient(hunting): my global_id is 4340 2024-03-25T19:22:09.267 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.267+0000 7fea4f7fe640 1 -- 172.21.15.83:0/3302568104 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea58136ad0 con 0x7fea5813f710 2024-03-25T19:22:09.272 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.272+0000 7fea4f7fe640 1 -- 172.21.15.83:0/3302568104 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1152483523 0 0) 0x7fea580b7520 con 0x7fea581430c0 2024-03-25T19:22:09.272 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.272+0000 7fea4f7fe640 1 -- 172.21.15.83:0/3302568104 >> v1:172.21.15.83:6790/0 conn(0x7fea5813f710 legacy=0x7fea580b5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:09.273 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.272+0000 7fea4f7fe640 1 -- 172.21.15.83:0/3302568104 >> v1:172.21.15.103:6789/0 conn(0x7fea580a98f0 legacy=0x7fea58135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:09.273 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.272+0000 7fea4f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:22:09.273 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.272+0000 7fea4f7fe640 1 monclient: found mon.a 2024-03-25T19:22:09.273 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.272+0000 7fea4f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:09.273 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.272+0000 7fea4f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:22:09.273 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.272+0000 7fea4f7fe640 1 -- 172.21.15.83:0/3302568104 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea580b6180 con 0x7fea581430c0 2024-03-25T19:22:09.273 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.272+0000 7fea4f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:22:09.273 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.272+0000 7fea4f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:22:09.273 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.272+0000 7fea4f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:09.273 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.272+0000 7fea5d132640 5 monclient: authenticate success, global_id 4329 2024-03-25T19:22:09.273 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.272+0000 7fea5d132640 10 monclient: _renew_subs 2024-03-25T19:22:09.273 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.272+0000 7fea5d132640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:22:09.273 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.272+0000 7fea5d132640 1 -- 172.21.15.83:0/3302568104 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea580b7a40 con 0x7fea581430c0 2024-03-25T19:22:09.273 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.272+0000 7fea5d132640 10 client.4329.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:22:09.273 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.272+0000 7fea5d132640 10 monclient: _renew_subs 2024-03-25T19:22:09.274 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.272+0000 7fea5d132640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:22:09.274 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.272+0000 7fea5d132640 1 -- 172.21.15.83:0/3302568104 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fea58131840 con 0x7fea581430c0 2024-03-25T19:22:09.274 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.272+0000 7fea5d132640 1 librados: init done 2024-03-25T19:22:09.274 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.273+0000 7fea5d132640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:09.274 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.273+0000 7fea5d132640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:09.274 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.273+0000 7fea5d132640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:22:09.274 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.273+0000 7fea5d132640 1 -- 172.21.15.83:0/3302568104 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea58136f60 con 0x7fea581430c0 2024-03-25T19:22:09.578 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.578+0000 7fea4f7fe640 1 -- 172.21.15.83:0/3302568104 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea50004e10 con 0x7fea581430c0 2024-03-25T19:22:09.578 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.578+0000 7fea4f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:09.578 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.578+0000 7fea4f7fe640 1 -- 172.21.15.83:0/3302568104 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fea50004f80 con 0x7fea581430c0 2024-03-25T19:22:09.579 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.578+0000 7fea4f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:09.579 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.578+0000 7fea4f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:22:09.579 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.578+0000 7fea4f7fe640 10 monclient: dump: 2024-03-25T19:22:09.579 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:22:09.579 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:22:09.579 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:09.579 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:09.579 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:09.579 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:22:09.579 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:22:09.579 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:22:09.579 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:22:09.579 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:09.579 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.578+0000 7fea55d9b640 4 set_mon_vals no callback set 2024-03-25T19:22:09.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.579+0000 7fea4f7fe640 1 -- 172.21.15.83:0/3302568104 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7fea5001af50 con 0x7fea581430c0 2024-03-25T19:22:09.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.580+0000 7fea56d9d640 10 monclient: get_auth_request con 0x7fea3806df00 auth_method 0 2024-03-25T19:22:09.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.580+0000 7fea4f7fe640 10 client.4329.objecter ms_handle_connect 0x7fea3806df00 2024-03-25T19:22:09.600 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.600+0000 7fea4f7fe640 1 -- 172.21.15.83:0/3302568104 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7fea50054a70 con 0x7fea581430c0 2024-03-25T19:22:09.600 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.600+0000 7fea4f7fe640 10 client.4329.objecter ms_dispatch 0x7fea58000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:22:09.600 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.600+0000 7fea4f7fe640 3 client.4329.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:22:09.600 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.600+0000 7fea4f7fe640 3 client.4329.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:22:09.600 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.600+0000 7fea4f7fe640 20 client.4329.objecter dump_active .. 0 homeless 2024-03-25T19:22:09.600 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.600+0000 7fea4f7fe640 1 -- 172.21.15.83:0/3302568104 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fea50050580 con 0x7fea581430c0 2024-03-25T19:22:09.600 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.600+0000 7fea4f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:09.600 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.600+0000 7fea4f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:22:09.846 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.846+0000 7fea5d132640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:09.846 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.846+0000 7fea5d132640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:09.846 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.846+0000 7fea5d132640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:22:09.846 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.846+0000 7fea5d132640 1 -- 172.21.15.83:0/3302568104 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fea580b6310 con 0x7fea581430c0 2024-03-25T19:22:09.872 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.872+0000 7fea4f7fe640 1 -- 172.21.15.83:0/3302568104 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7fea500524b0 con 0x7fea581430c0 2024-03-25T19:22:09.872 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.872+0000 7fea4f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:09.872 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.872+0000 7fea4f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:22:09.872 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:22:09.872 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:22:09.875 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.874+0000 7fea5d132640 10 librados: watch_flush enter 2024-03-25T19:22:09.875 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.874+0000 7fea5d132640 10 librados: watch_flush exit 2024-03-25T19:22:09.875 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.874+0000 7fea5d132640 20 client.4329.objecter shutdown clearing up homeless session... 2024-03-25T19:22:09.875 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.874+0000 7fea5d132640 10 client.4329.objecter successfully canceled tick 2024-03-25T19:22:09.875 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.874+0000 7fea5d132640 1 -- 172.21.15.83:0/3302568104 >> v1:172.21.15.103:6800/345072327 conn(0x7fea3806df00 legacy=0x7fea380703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:09.875 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.874+0000 7fea5d132640 10 monclient: shutdown 2024-03-25T19:22:09.875 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.874+0000 7fea5d132640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:09.875 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.874+0000 7fea5d132640 1 -- 172.21.15.83:0/3302568104 >> v1:172.21.15.83:6789/0 conn(0x7fea581430c0 legacy=0x7fea58130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:09.876 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.874+0000 7fea5d132640 1 -- 172.21.15.83:0/3302568104 shutdown_connections 2024-03-25T19:22:09.876 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.874+0000 7fea5d132640 1 -- 172.21.15.83:0/3302568104 >> 172.21.15.83:0/3302568104 conn(0x7fea580b7ea0 msgr2=0x7fea58074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:09.876 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.875+0000 7fea5d132640 1 -- 172.21.15.83:0/3302568104 shutdown_connections 2024-03-25T19:22:09.876 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.875+0000 7fea5d132640 1 -- 172.21.15.83:0/3302568104 wait complete. 2024-03-25T19:22:09.876 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:09.875+0000 7fea5d132640 1 librados: shutdown 2024-03-25T19:22:09.885 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:22:16.691 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:22:16.805 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.806+0000 7f134170e640 10 monclient: get_monmap_and_config 2024-03-25T19:22:16.806 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.806+0000 7f134170e640 10 monclient: build_initial_monmap 2024-03-25T19:22:16.806 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.806+0000 7f134170e640 10 monclient: monmap: 2024-03-25T19:22:16.806 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:22:16.806 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:22:16.806 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:22:16.806970+0000 2024-03-25T19:22:16.806 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:22:16.806970+0000 2024-03-25T19:22:16.806 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:16.806 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:22:16.806 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:22:16.806 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:22:16.806 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:22:16.806 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:16.808 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.806+0000 7f134170e640 1 Processor -- start 2024-03-25T19:22:16.808 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.806+0000 7f134170e640 1 -- start start 2024-03-25T19:22:16.808 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.806+0000 7f134170e640 10 monclient: init 2024-03-25T19:22:16.808 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.807+0000 7f134170e640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:16.808 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.807+0000 7f134170e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:22:16.808 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.807+0000 7f134170e640 10 monclient(hunting): picked mon.noname-b con 0x7f133c0a98f0 addr v1:172.21.15.83:6790/0 2024-03-25T19:22:16.808 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.807+0000 7f134170e640 10 monclient(hunting): picked mon.noname-a con 0x7f133c13f710 addr v1:172.21.15.83:6789/0 2024-03-25T19:22:16.808 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.807+0000 7f134170e640 10 monclient(hunting): picked mon.noname-c con 0x7f133c1430c0 addr v1:172.21.15.103:6789/0 2024-03-25T19:22:16.808 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.807+0000 7f134170e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f133c142520 con 0x7f133c13f710 2024-03-25T19:22:16.808 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.807+0000 7f134170e640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f133c0b28a0 con 0x7f133c1430c0 2024-03-25T19:22:16.808 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.807+0000 7f134170e640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f133c0b2a30 con 0x7f133c0a98f0 2024-03-25T19:22:16.808 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.807+0000 7f134170e640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:16.808 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.807+0000 7f134170e640 10 monclient(hunting): authenticate will time out at 6194.079598s 2024-03-25T19:22:16.817 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.818+0000 7f133b7fe640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f133c1430c0 0x7f133c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:58106/0 (socket says 172.21.15.83:58106) 2024-03-25T19:22:16.817 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.818+0000 7f133b7fe640 1 -- 172.21.15.83:0/1251371755 learned_addr learned my addr 172.21.15.83:0/1251371755 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:22:16.827 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.828+0000 7f1339ffb640 1 -- 172.21.15.83:0/1251371755 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3125304197 0 0) 0x7f133c0b28a0 con 0x7f133c1430c0 2024-03-25T19:22:16.828 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.828+0000 7f1339ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:22:16.828 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.828+0000 7f1339ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:16.828 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.828+0000 7f1339ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:16.828 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.828+0000 7f1339ffb640 10 monclient(hunting): my global_id is 4336 2024-03-25T19:22:16.828 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.828+0000 7f1339ffb640 1 -- 172.21.15.83:0/1251371755 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1328003520 con 0x7f133c1430c0 2024-03-25T19:22:16.832 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.832+0000 7f1339ffb640 1 -- 172.21.15.83:0/1251371755 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4210224583 0 0) 0x7f133c142520 con 0x7f133c13f710 2024-03-25T19:22:16.832 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.832+0000 7f1339ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:22:16.832 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.832+0000 7f1339ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:16.832 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.832+0000 7f1339ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:16.832 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.832+0000 7f1339ffb640 10 monclient(hunting): my global_id is 4332 2024-03-25T19:22:16.832 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.832+0000 7f1339ffb640 1 -- 172.21.15.83:0/1251371755 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f133c0b28a0 con 0x7f133c13f710 2024-03-25T19:22:16.844 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.844+0000 7f1339ffb640 1 -- 172.21.15.83:0/1251371755 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3576559230 0 0) 0x7f133c0b2a30 con 0x7f133c0a98f0 2024-03-25T19:22:16.844 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.844+0000 7f1339ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:22:16.844 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.844+0000 7f1339ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:16.844 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.844+0000 7f1339ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:16.844 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.844+0000 7f1339ffb640 10 monclient(hunting): my global_id is 4343 2024-03-25T19:22:16.844 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.844+0000 7f1339ffb640 1 -- 172.21.15.83:0/1251371755 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f133c142520 con 0x7f133c0a98f0 2024-03-25T19:22:16.845 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.845+0000 7f1339ffb640 1 -- 172.21.15.83:0/1251371755 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2623168995 0 0) 0x7f133c0b28a0 con 0x7f133c13f710 2024-03-25T19:22:16.845 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.845+0000 7f1339ffb640 1 -- 172.21.15.83:0/1251371755 >> v1:172.21.15.83:6790/0 conn(0x7f133c0a98f0 legacy=0x7f133c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:16.845 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.845+0000 7f1339ffb640 1 -- 172.21.15.83:0/1251371755 >> v1:172.21.15.103:6789/0 conn(0x7f133c1430c0 legacy=0x7f133c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:16.845 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.845+0000 7f1339ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:22:16.845 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.845+0000 7f1339ffb640 1 monclient: found mon.noname-a 2024-03-25T19:22:16.845 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.845+0000 7f1339ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:16.845 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.845+0000 7f1339ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.83:6789/0 2024-03-25T19:22:16.846 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.845+0000 7f1339ffb640 1 -- 172.21.15.83:0/1251371755 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f133c0b44c0 con 0x7f133c13f710 2024-03-25T19:22:16.846 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.845+0000 7f1339ffb640 10 monclient: _finish_auth 0 2024-03-25T19:22:16.846 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.845+0000 7f1339ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:22:16.846 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.845+0000 7f1339ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:16.846 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.845+0000 7f1339ffb640 1 -- 172.21.15.83:0/1251371755 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f1324001670 con 0x7f133c13f710 2024-03-25T19:22:16.846 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.845+0000 7f1339ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:16.846 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.845+0000 7f1339ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:22:16.846 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.845+0000 7f1339ffb640 10 monclient: dump: 2024-03-25T19:22:16.846 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:22:16.846 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:22:16.846 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:16.846 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:16.846 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:16.846 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:22:16.846 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:22:16.847 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:22:16.847 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:22:16.847 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:16.847 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.845+0000 7f134170e640 5 monclient: authenticate success, global_id 4332 2024-03-25T19:22:16.847 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.845+0000 7f134170e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:16.858 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.859+0000 7f1339ffb640 1 -- 172.21.15.83:0/1251371755 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13240017d0 con 0x7f133c13f710 2024-03-25T19:22:16.858 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.859+0000 7f1339ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:16.859 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.859+0000 7f1339ffb640 1 -- 172.21.15.83:0/1251371755 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f1324001be0 con 0x7f133c13f710 2024-03-25T19:22:16.859 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.859+0000 7f134170e640 10 monclient: get_monmap_and_config success 2024-03-25T19:22:16.859 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.859+0000 7f134170e640 4 set_mon_vals no callback set 2024-03-25T19:22:16.859 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.859+0000 7f1339ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:16.859 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.859+0000 7f1339ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:22:16.859 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.859+0000 7f1339ffb640 10 monclient: dump: 2024-03-25T19:22:16.859 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:22:16.859 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:22:16.859 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:16.859 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:16.859 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:16.859 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:22:16.859 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:22:16.860 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:22:16.860 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:22:16.860 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:16.861 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.860+0000 7f134170e640 10 monclient: shutdown 2024-03-25T19:22:16.862 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.860+0000 7f134170e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:16.862 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.860+0000 7f134170e640 1 -- 172.21.15.83:0/1251371755 >> v1:172.21.15.83:6789/0 conn(0x7f133c13f710 legacy=0x7f133c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:16.862 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.860+0000 7f134170e640 1 -- 172.21.15.83:0/1251371755 shutdown_connections 2024-03-25T19:22:16.862 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.860+0000 7f134170e640 1 -- 172.21.15.83:0/1251371755 >> 172.21.15.83:0/1251371755 conn(0x7f133c0b7ea0 msgr2=0x7f133c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:16.862 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.860+0000 7f134170e640 1 -- 172.21.15.83:0/1251371755 shutdown_connections 2024-03-25T19:22:16.862 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.860+0000 7f134170e640 1 -- 172.21.15.83:0/1251371755 wait complete. 2024-03-25T19:22:16.862 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.861+0000 7f134170e640 10 monclient: build_initial_monmap 2024-03-25T19:22:16.862 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.861+0000 7f134170e640 10 monclient: monmap: 2024-03-25T19:22:16.862 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:22:16.862 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:22:16.862 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:22:16.862 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:22:16.862 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:16.862 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:22:16.863 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:22:16.863 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:22:16.863 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:22:16.863 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:16.863 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.861+0000 7f134170e640 1 librados: starting msgr at 2024-03-25T19:22:16.863 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.861+0000 7f134170e640 1 librados: starting objecter 2024-03-25T19:22:16.863 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.861+0000 7f134170e640 1 Processor -- start 2024-03-25T19:22:16.863 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.861+0000 7f134170e640 1 -- start start 2024-03-25T19:22:16.863 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.861+0000 7f134170e640 1 librados: setting wanted keys 2024-03-25T19:22:16.863 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.861+0000 7f134170e640 1 librados: calling monclient init 2024-03-25T19:22:16.863 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.861+0000 7f134170e640 10 monclient: init 2024-03-25T19:22:16.863 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.862+0000 7f134170e640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:16.863 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.862+0000 7f134170e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:22:16.863 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.862+0000 7f134170e640 10 monclient(hunting): picked mon.noname-c con 0x7f133c0a98f0 addr v1:172.21.15.83:6790/0 2024-03-25T19:22:16.864 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.862+0000 7f134170e640 10 monclient(hunting): picked mon.noname-b con 0x7f133c13f710 addr v1:172.21.15.103:6789/0 2024-03-25T19:22:16.864 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.862+0000 7f134170e640 10 monclient(hunting): picked mon.noname-a con 0x7f133c1430c0 addr v1:172.21.15.83:6789/0 2024-03-25T19:22:16.864 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.862+0000 7f134170e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f133c136750 con 0x7f133c1430c0 2024-03-25T19:22:16.864 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.862+0000 7f134170e640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f133c1dac30 con 0x7f133c13f710 2024-03-25T19:22:16.864 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.862+0000 7f134170e640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f133c1399c0 con 0x7f133c0a98f0 2024-03-25T19:22:16.864 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.862+0000 7f134170e640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:16.864 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.862+0000 7f134170e640 10 monclient(hunting): authenticate will time out at 6194.134277s 2024-03-25T19:22:16.873 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.874+0000 7f133a7fc640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f133c13f710 0x7f133c1db150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:58158/0 (socket says 172.21.15.83:58158) 2024-03-25T19:22:16.873 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.874+0000 7f133a7fc640 1 -- 172.21.15.83:0/1312818040 learned_addr learned my addr 172.21.15.83:0/1312818040 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:22:16.877 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.877+0000 7f13337fe640 10 client.?.objecter ms_handle_connect 0x7f133c13f710 2024-03-25T19:22:16.877 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.877+0000 7f13337fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:16.879 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.879+0000 7f13337fe640 10 client.?.objecter ms_handle_connect 0x7f133c0a98f0 2024-03-25T19:22:16.879 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.879+0000 7f13337fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:16.879 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.879+0000 7f13337fe640 10 client.?.objecter ms_handle_connect 0x7f133c1430c0 2024-03-25T19:22:16.879 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.879+0000 7f13337fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:16.891 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.891+0000 7f13337fe640 1 -- 172.21.15.83:0/1312818040 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2431160958 0 0) 0x7f133c136750 con 0x7f133c1430c0 2024-03-25T19:22:16.891 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.891+0000 7f13337fe640 10 monclient(hunting): _negotiate 2024-03-25T19:22:16.891 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.891+0000 7f13337fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:16.891 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.891+0000 7f13337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:16.891 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.891+0000 7f13337fe640 10 monclient(hunting): my global_id is 4335 2024-03-25T19:22:16.891 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.891+0000 7f13337fe640 1 -- 172.21.15.83:0/1312818040 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f131c004eb0 con 0x7f133c1430c0 2024-03-25T19:22:16.904 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.904+0000 7f13337fe640 1 -- 172.21.15.83:0/1312818040 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3005780854 0 0) 0x7f133c1399c0 con 0x7f133c0a98f0 2024-03-25T19:22:16.904 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.904+0000 7f13337fe640 10 monclient(hunting): _negotiate 2024-03-25T19:22:16.904 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.904+0000 7f13337fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:16.904 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.904+0000 7f13337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:16.904 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.904+0000 7f13337fe640 10 monclient(hunting): my global_id is 4346 2024-03-25T19:22:16.904 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.904+0000 7f13337fe640 1 -- 172.21.15.83:0/1312818040 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f133c136750 con 0x7f133c0a98f0 2024-03-25T19:22:16.904 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.904+0000 7f13337fe640 1 -- 172.21.15.83:0/1312818040 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 745200727 0 0) 0x7f133c1dac30 con 0x7f133c13f710 2024-03-25T19:22:16.904 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.904+0000 7f13337fe640 10 monclient(hunting): _negotiate 2024-03-25T19:22:16.905 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.904+0000 7f13337fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:16.905 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.904+0000 7f13337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:16.905 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.904+0000 7f13337fe640 10 monclient(hunting): my global_id is 4339 2024-03-25T19:22:16.905 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.904+0000 7f13337fe640 1 -- 172.21.15.83:0/1312818040 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f133c1399c0 con 0x7f133c13f710 2024-03-25T19:22:16.911 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.911+0000 7f13337fe640 1 -- 172.21.15.83:0/1312818040 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2246748747 0 0) 0x7f131c004eb0 con 0x7f133c1430c0 2024-03-25T19:22:16.911 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.911+0000 7f13337fe640 1 -- 172.21.15.83:0/1312818040 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f133c1dac30 con 0x7f133c1430c0 2024-03-25T19:22:16.911 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.911+0000 7f13337fe640 1 -- 172.21.15.83:0/1312818040 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f1334001af0 con 0x7f133c1430c0 2024-03-25T19:22:16.911 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.911+0000 7f13337fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:16.911 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.911+0000 7f13337fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:22:16.911 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.911+0000 7f13337fe640 10 monclient(hunting): dump: 2024-03-25T19:22:16.911 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:22:16.911 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:22:16.911 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:16.912 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:16.912 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:16.912 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:22:16.912 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:22:16.912 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:22:16.912 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:22:16.912 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:16.912 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.911+0000 7f13337fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:22:16.919 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.919+0000 7f13337fe640 1 -- 172.21.15.83:0/1312818040 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2838834144 0 0) 0x7f133c1399c0 con 0x7f133c13f710 2024-03-25T19:22:16.919 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.919+0000 7f13337fe640 1 -- 172.21.15.83:0/1312818040 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f131c004eb0 con 0x7f133c13f710 2024-03-25T19:22:16.919 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.919+0000 7f13337fe640 1 -- 172.21.15.83:0/1312818040 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f1324001630 con 0x7f133c13f710 2024-03-25T19:22:16.919 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.919+0000 7f13337fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:16.919 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.919+0000 7f13337fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:22:16.919 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.919+0000 7f13337fe640 10 monclient(hunting): dump: 2024-03-25T19:22:16.919 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:22:16.919 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:22:16.919 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:16.919 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:16.919 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:16.920 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:22:16.920 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:22:16.920 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:22:16.920 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:22:16.920 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:16.925 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.925+0000 7f13337fe640 1 -- 172.21.15.83:0/1312818040 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 538793151 0 0) 0x7f133c1dac30 con 0x7f133c1430c0 2024-03-25T19:22:16.925 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.925+0000 7f13337fe640 1 -- 172.21.15.83:0/1312818040 >> v1:172.21.15.83:6790/0 conn(0x7f133c0a98f0 legacy=0x7f133c1386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:16.925 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.925+0000 7f13337fe640 1 -- 172.21.15.83:0/1312818040 >> v1:172.21.15.103:6789/0 conn(0x7f133c13f710 legacy=0x7f133c1db150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:16.925 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.925+0000 7f13337fe640 10 monclient: _finish_hunting 0 2024-03-25T19:22:16.925 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.925+0000 7f13337fe640 1 monclient: found mon.a 2024-03-25T19:22:16.925 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.925+0000 7f13337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:16.925 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.925+0000 7f13337fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:22:16.926 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.925+0000 7f13337fe640 1 -- 172.21.15.83:0/1312818040 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f133c1db860 con 0x7f133c1430c0 2024-03-25T19:22:16.926 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.925+0000 7f13337fe640 10 monclient: _finish_auth 0 2024-03-25T19:22:16.926 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.925+0000 7f13337fe640 10 monclient: _check_auth_tickets 2024-03-25T19:22:16.926 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.925+0000 7f13337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:16.926 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.925+0000 7f134170e640 5 monclient: authenticate success, global_id 4335 2024-03-25T19:22:16.926 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.925+0000 7f134170e640 10 monclient: _renew_subs 2024-03-25T19:22:16.926 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.925+0000 7f134170e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:22:16.926 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.925+0000 7f134170e640 1 -- 172.21.15.83:0/1312818040 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f133c1d41e0 con 0x7f133c1430c0 2024-03-25T19:22:16.926 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.925+0000 7f134170e640 10 client.4335.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:22:16.926 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.925+0000 7f134170e640 10 monclient: _renew_subs 2024-03-25T19:22:16.926 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.925+0000 7f134170e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:22:16.926 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.925+0000 7f134170e640 1 -- 172.21.15.83:0/1312818040 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f133c1d5150 con 0x7f133c1430c0 2024-03-25T19:22:16.926 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.925+0000 7f134170e640 1 librados: init done 2024-03-25T19:22:16.927 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.926+0000 7f134170e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:16.927 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.926+0000 7f134170e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:16.927 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.926+0000 7f134170e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:22:16.927 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:16.926+0000 7f134170e640 1 -- 172.21.15.83:0/1312818040 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f133c1daa90 con 0x7f133c1430c0 2024-03-25T19:22:17.238 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:17.238+0000 7f13337fe640 1 -- 172.21.15.83:0/1312818040 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1334001f20 con 0x7f133c1430c0 2024-03-25T19:22:17.238 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:17.238+0000 7f13337fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:17.238 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:17.238+0000 7f13337fe640 1 -- 172.21.15.83:0/1312818040 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f1334004b60 con 0x7f133c1430c0 2024-03-25T19:22:17.238 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:17.238+0000 7f13337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:17.239 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:17.238+0000 7f13337fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:22:17.239 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:17.238+0000 7f13337fe640 10 monclient: dump: 2024-03-25T19:22:17.239 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:22:17.239 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:22:17.239 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:17.239 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:17.239 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:17.239 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:22:17.239 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:22:17.239 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:22:17.239 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:22:17.239 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:17.240 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:17.238+0000 7f13397fa640 4 set_mon_vals no callback set 2024-03-25T19:22:17.240 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:17.239+0000 7f13337fe640 1 -- 172.21.15.83:0/1312818040 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7f1334004d40 con 0x7f133c1430c0 2024-03-25T19:22:17.240 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:17.240+0000 7f133affd640 10 monclient: get_auth_request con 0x7f131c06df70 auth_method 0 2024-03-25T19:22:17.240 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:17.240+0000 7f13337fe640 10 client.4335.objecter ms_handle_connect 0x7f131c06df70 2024-03-25T19:22:17.244 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:17.245+0000 7f13337fe640 1 -- 172.21.15.83:0/1312818040 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7f1334054890 con 0x7f133c1430c0 2024-03-25T19:22:17.244 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:17.245+0000 7f13337fe640 10 client.4335.objecter ms_dispatch 0x7f133c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:22:17.244 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:17.245+0000 7f13337fe640 3 client.4335.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:22:17.245 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:17.245+0000 7f13337fe640 3 client.4335.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:22:17.245 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:17.245+0000 7f13337fe640 20 client.4335.objecter dump_active .. 0 homeless 2024-03-25T19:22:17.245 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:17.245+0000 7f13337fe640 1 -- 172.21.15.83:0/1312818040 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f13340503a0 con 0x7f133c1430c0 2024-03-25T19:22:17.245 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:17.245+0000 7f13337fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:17.245 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:17.245+0000 7f13337fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:22:17.444 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:17.444+0000 7f134170e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:17.444 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:17.444+0000 7f134170e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:17.444 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:17.444+0000 7f134170e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:22:17.444 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:17.444+0000 7f134170e640 1 -- 172.21.15.83:0/1312818040 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f133c1db9f0 con 0x7f133c1430c0 2024-03-25T19:22:17.483 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:17.483+0000 7f13337fe640 1 -- 172.21.15.83:0/1312818040 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7f133c1db9f0 con 0x7f133c1430c0 2024-03-25T19:22:17.483 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:17.483+0000 7f13337fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:17.483 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:17.483+0000 7f13337fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:22:17.483 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:22:17.483 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:22:17.486 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:17.485+0000 7f134170e640 10 librados: watch_flush enter 2024-03-25T19:22:17.486 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:17.485+0000 7f134170e640 10 librados: watch_flush exit 2024-03-25T19:22:17.486 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:17.485+0000 7f134170e640 20 client.4335.objecter shutdown clearing up homeless session... 2024-03-25T19:22:17.486 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:17.485+0000 7f134170e640 10 client.4335.objecter successfully canceled tick 2024-03-25T19:22:17.486 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:17.485+0000 7f134170e640 1 -- 172.21.15.83:0/1312818040 >> v1:172.21.15.103:6800/345072327 conn(0x7f131c06df70 legacy=0x7f131c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:17.486 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:17.485+0000 7f134170e640 10 monclient: shutdown 2024-03-25T19:22:17.486 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:17.485+0000 7f134170e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:17.486 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:17.485+0000 7f134170e640 1 -- 172.21.15.83:0/1312818040 >> v1:172.21.15.83:6789/0 conn(0x7f133c1430c0 legacy=0x7f133c1d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:17.486 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:17.485+0000 7f134170e640 1 -- 172.21.15.83:0/1312818040 shutdown_connections 2024-03-25T19:22:17.486 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:17.485+0000 7f134170e640 1 -- 172.21.15.83:0/1312818040 >> 172.21.15.83:0/1312818040 conn(0x7f133c0b7ea0 msgr2=0x7f133c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:17.486 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:17.485+0000 7f134170e640 1 -- 172.21.15.83:0/1312818040 shutdown_connections 2024-03-25T19:22:17.487 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:17.485+0000 7f134170e640 1 -- 172.21.15.83:0/1312818040 wait complete. 2024-03-25T19:22:17.487 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:17.485+0000 7f134170e640 1 librados: shutdown 2024-03-25T19:22:17.496 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:22:22.562 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:03:04.489 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:22.570 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:03:04.497 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:22.609 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:03:04.536 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:22.615 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:03:04.525 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:22.622 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:03:04.532 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:22.633 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:03:04.561 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:22.639 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:03:04.566 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:22.641 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:03:04.551 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:22.671 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:03:04.599 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:24.302 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:22:24.417 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.417+0000 7f87bdf8c640 10 monclient: get_monmap_and_config 2024-03-25T19:22:24.417 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.417+0000 7f87bdf8c640 10 monclient: build_initial_monmap 2024-03-25T19:22:24.417 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.417+0000 7f87bdf8c640 10 monclient: monmap: 2024-03-25T19:22:24.417 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:22:24.417 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:22:24.417 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:22:24.418242+0000 2024-03-25T19:22:24.417 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:22:24.418242+0000 2024-03-25T19:22:24.417 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:24.417 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:22:24.417 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:22:24.418 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:22:24.418 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:22:24.418 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:24.419 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.418+0000 7f87bdf8c640 1 Processor -- start 2024-03-25T19:22:24.419 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.418+0000 7f87bdf8c640 1 -- start start 2024-03-25T19:22:24.419 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.418+0000 7f87bdf8c640 10 monclient: init 2024-03-25T19:22:24.419 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.419+0000 7f87bdf8c640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:24.420 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.419+0000 7f87bdf8c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:22:24.420 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.419+0000 7f87bdf8c640 10 monclient(hunting): picked mon.noname-a con 0x7f87b80a98f0 addr v1:172.21.15.83:6789/0 2024-03-25T19:22:24.420 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.419+0000 7f87bdf8c640 10 monclient(hunting): picked mon.noname-c con 0x7f87b813f710 addr v1:172.21.15.103:6789/0 2024-03-25T19:22:24.420 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.419+0000 7f87bdf8c640 10 monclient(hunting): picked mon.noname-b con 0x7f87b81430c0 addr v1:172.21.15.83:6790/0 2024-03-25T19:22:24.420 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.419+0000 7f87bdf8c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87b8142520 con 0x7f87b80a98f0 2024-03-25T19:22:24.420 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.419+0000 7f87bdf8c640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87b80b28a0 con 0x7f87b813f710 2024-03-25T19:22:24.420 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.419+0000 7f87bdf8c640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87b80b2a30 con 0x7f87b81430c0 2024-03-25T19:22:24.420 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.419+0000 7f87bdf8c640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:24.420 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.419+0000 7f87bdf8c640 10 monclient(hunting): authenticate will time out at 6201.691408s 2024-03-25T19:22:24.421 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.422+0000 7f87b7fff640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f87b81430c0 0x7f87b814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:40276/0 (socket says 172.21.15.83:40276) 2024-03-25T19:22:24.422 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.422+0000 7f87b7fff640 1 -- 172.21.15.83:0/176098029 learned_addr learned my addr 172.21.15.83:0/176098029 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:22:24.437 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.437+0000 7f87b67fc640 1 -- 172.21.15.83:0/176098029 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2203847998 0 0) 0x7f87b80b2a30 con 0x7f87b81430c0 2024-03-25T19:22:24.437 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.437+0000 7f87b67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:22:24.437 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.437+0000 7f87b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:24.437 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.437+0000 7f87b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:24.437 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.437+0000 7f87b67fc640 10 monclient(hunting): my global_id is 4349 2024-03-25T19:22:24.437 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.437+0000 7f87b67fc640 1 -- 172.21.15.83:0/176098029 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87a0003520 con 0x7f87b81430c0 2024-03-25T19:22:24.445 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.446+0000 7f87b67fc640 1 -- 172.21.15.83:0/176098029 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2649099165 0 0) 0x7f87b8142520 con 0x7f87b80a98f0 2024-03-25T19:22:24.446 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.446+0000 7f87b67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:22:24.446 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.446+0000 7f87b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:24.446 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.446+0000 7f87b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:24.446 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.446+0000 7f87b67fc640 10 monclient(hunting): my global_id is 4338 2024-03-25T19:22:24.446 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.446+0000 7f87b67fc640 1 -- 172.21.15.83:0/176098029 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87b80b2a30 con 0x7f87b80a98f0 2024-03-25T19:22:24.452 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.452+0000 7f87b67fc640 1 -- 172.21.15.83:0/176098029 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3265849937 0 0) 0x7f87a0003520 con 0x7f87b81430c0 2024-03-25T19:22:24.453 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.452+0000 7f87b67fc640 1 -- 172.21.15.83:0/176098029 >> v1:172.21.15.103:6789/0 conn(0x7f87b813f710 legacy=0x7f87b81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:24.453 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.452+0000 7f87b67fc640 1 -- 172.21.15.83:0/176098029 >> v1:172.21.15.83:6789/0 conn(0x7f87b80a98f0 legacy=0x7f87b80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:24.453 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.452+0000 7f87b67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:22:24.453 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.452+0000 7f87b67fc640 1 monclient: found mon.noname-b 2024-03-25T19:22:24.453 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.452+0000 7f87b67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:24.453 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.452+0000 7f87b67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.83:6790/0 2024-03-25T19:22:24.453 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.452+0000 7f87b67fc640 1 -- 172.21.15.83:0/176098029 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87b80b44c0 con 0x7f87b81430c0 2024-03-25T19:22:24.453 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.452+0000 7f87b67fc640 10 monclient: _finish_auth 0 2024-03-25T19:22:24.453 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.452+0000 7f87b67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:22:24.453 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.452+0000 7f87b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:24.453 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.453+0000 7f87b67fc640 1 -- 172.21.15.83:0/176098029 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f87b0001a10 con 0x7f87b81430c0 2024-03-25T19:22:24.453 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.453+0000 7f87bdf8c640 5 monclient: authenticate success, global_id 4349 2024-03-25T19:22:24.453 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.453+0000 7f87bdf8c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:24.453 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.453+0000 7f87b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:24.454 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.453+0000 7f87b67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:22:24.454 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.453+0000 7f87b67fc640 10 monclient: dump: 2024-03-25T19:22:24.454 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:22:24.454 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:22:24.454 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:24.454 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:24.454 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:24.454 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:22:24.454 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:22:24.454 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:22:24.454 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:22:24.454 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:24.454 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.453+0000 7f87bdf8c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:24.473 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.473+0000 7f87b67fc640 1 -- 172.21.15.83:0/176098029 <== mon.2 v1:172.21.15.83:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f87b0001b70 con 0x7f87b81430c0 2024-03-25T19:22:24.473 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.473+0000 7f87b67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:24.473 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.473+0000 7f87b67fc640 1 -- 172.21.15.83:0/176098029 <== mon.2 v1:172.21.15.83:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f87b0001f80 con 0x7f87b81430c0 2024-03-25T19:22:24.473 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.473+0000 7f87b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:24.473 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.473+0000 7f87b67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:22:24.473 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.473+0000 7f87b67fc640 10 monclient: dump: 2024-03-25T19:22:24.473 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:22:24.474 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:22:24.474 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:24.474 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:24.474 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:24.474 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:22:24.474 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:22:24.474 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:22:24.474 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:22:24.474 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:24.474 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.473+0000 7f87bdf8c640 10 monclient: get_monmap_and_config success 2024-03-25T19:22:24.474 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.473+0000 7f87bdf8c640 4 set_mon_vals no callback set 2024-03-25T19:22:24.476 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.474+0000 7f87bdf8c640 10 monclient: shutdown 2024-03-25T19:22:24.477 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.474+0000 7f87bdf8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:24.477 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.474+0000 7f87bdf8c640 1 -- 172.21.15.83:0/176098029 >> v1:172.21.15.83:6790/0 conn(0x7f87b81430c0 legacy=0x7f87b814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:24.477 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.474+0000 7f87bdf8c640 1 -- 172.21.15.83:0/176098029 shutdown_connections 2024-03-25T19:22:24.477 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.474+0000 7f87bdf8c640 1 -- 172.21.15.83:0/176098029 >> 172.21.15.83:0/176098029 conn(0x7f87b80b7ea0 msgr2=0x7f87b80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:24.477 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.474+0000 7f87bdf8c640 1 -- 172.21.15.83:0/176098029 shutdown_connections 2024-03-25T19:22:24.477 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.474+0000 7f87bdf8c640 1 -- 172.21.15.83:0/176098029 wait complete. 2024-03-25T19:22:24.477 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.475+0000 7f87bdf8c640 10 monclient: build_initial_monmap 2024-03-25T19:22:24.477 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.475+0000 7f87bdf8c640 10 monclient: monmap: 2024-03-25T19:22:24.477 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:22:24.477 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:22:24.477 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:22:24.477 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:22:24.477 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:24.477 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:22:24.477 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:22:24.478 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:22:24.478 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:22:24.478 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:24.478 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.475+0000 7f87bdf8c640 1 librados: starting msgr at 2024-03-25T19:22:24.478 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.475+0000 7f87bdf8c640 1 librados: starting objecter 2024-03-25T19:22:24.478 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.475+0000 7f87bdf8c640 1 Processor -- start 2024-03-25T19:22:24.478 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.476+0000 7f87bdf8c640 1 -- start start 2024-03-25T19:22:24.478 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.476+0000 7f87bdf8c640 1 librados: setting wanted keys 2024-03-25T19:22:24.478 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.476+0000 7f87bdf8c640 1 librados: calling monclient init 2024-03-25T19:22:24.478 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.476+0000 7f87bdf8c640 10 monclient: init 2024-03-25T19:22:24.478 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.476+0000 7f87bdf8c640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:24.479 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.476+0000 7f87bdf8c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:22:24.479 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.476+0000 7f87bdf8c640 10 monclient(hunting): picked mon.noname-a con 0x7f87b80a98f0 addr v1:172.21.15.83:6789/0 2024-03-25T19:22:24.479 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.476+0000 7f87bdf8c640 10 monclient(hunting): picked mon.noname-b con 0x7f87b813f710 addr v1:172.21.15.103:6789/0 2024-03-25T19:22:24.479 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.476+0000 7f87bdf8c640 10 monclient(hunting): picked mon.noname-c con 0x7f87b81430c0 addr v1:172.21.15.83:6790/0 2024-03-25T19:22:24.479 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.476+0000 7f87bdf8c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87b81daca0 con 0x7f87b80a98f0 2024-03-25T19:22:24.479 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.476+0000 7f87bdf8c640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87b8138ab0 con 0x7f87b813f710 2024-03-25T19:22:24.479 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.476+0000 7f87bdf8c640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87b80aba90 con 0x7f87b81430c0 2024-03-25T19:22:24.479 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.476+0000 7f87bdf8c640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:24.479 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.476+0000 7f87bdf8c640 10 monclient(hunting): authenticate will time out at 6201.748408s 2024-03-25T19:22:24.485 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.485+0000 7f87b6ffd640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f87b813f710 0x7f87b81db130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:34430/0 (socket says 172.21.15.83:34430) 2024-03-25T19:22:24.485 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.486+0000 7f87b6ffd640 1 -- 172.21.15.83:0/825497188 learned_addr learned my addr 172.21.15.83:0/825497188 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:22:24.487 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.488+0000 7f879ffff640 10 client.?.objecter ms_handle_connect 0x7f87b813f710 2024-03-25T19:22:24.488 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.488+0000 7f879ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:24.491 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.492+0000 7f879ffff640 10 client.?.objecter ms_handle_connect 0x7f87b81430c0 2024-03-25T19:22:24.491 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.492+0000 7f879ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:24.493 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.493+0000 7f879ffff640 1 -- 172.21.15.83:0/825497188 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2166178078 0 0) 0x7f87b8138ab0 con 0x7f87b813f710 2024-03-25T19:22:24.493 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.493+0000 7f879ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:22:24.493 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.493+0000 7f879ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:24.493 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.493+0000 7f879ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:24.493 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.493+0000 7f879ffff640 10 monclient(hunting): my global_id is 4345 2024-03-25T19:22:24.493 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.493+0000 7f879ffff640 1 -- 172.21.15.83:0/825497188 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87a4004eb0 con 0x7f87b813f710 2024-03-25T19:22:24.493 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.493+0000 7f879ffff640 10 client.?.objecter ms_handle_connect 0x7f87b80a98f0 2024-03-25T19:22:24.493 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.493+0000 7f879ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:24.501 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.501+0000 7f879ffff640 1 -- 172.21.15.83:0/825497188 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4125531066 0 0) 0x7f87a4004eb0 con 0x7f87b813f710 2024-03-25T19:22:24.501 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.502+0000 7f879ffff640 1 -- 172.21.15.83:0/825497188 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87b8138ab0 con 0x7f87b813f710 2024-03-25T19:22:24.502 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.502+0000 7f879ffff640 1 -- 172.21.15.83:0/825497188 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f87ac0019d0 con 0x7f87b813f710 2024-03-25T19:22:24.502 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.502+0000 7f879ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:24.502 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.502+0000 7f879ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:22:24.502 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.502+0000 7f879ffff640 10 monclient(hunting): dump: 2024-03-25T19:22:24.502 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:22:24.502 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:22:24.502 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:24.502 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:24.502 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:24.502 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:22:24.502 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:22:24.502 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:22:24.502 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:22:24.502 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:24.502 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.502+0000 7f879ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:22:24.503 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.503+0000 7f879ffff640 1 -- 172.21.15.83:0/825497188 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2600704442 0 0) 0x7f87b81daca0 con 0x7f87b80a98f0 2024-03-25T19:22:24.503 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.503+0000 7f879ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:22:24.503 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.503+0000 7f879ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:24.503 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.503+0000 7f879ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:24.503 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.503+0000 7f879ffff640 10 monclient(hunting): my global_id is 4341 2024-03-25T19:22:24.503 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.503+0000 7f879ffff640 1 -- 172.21.15.83:0/825497188 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87a4004eb0 con 0x7f87b80a98f0 2024-03-25T19:22:24.505 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.506+0000 7f879ffff640 1 -- 172.21.15.83:0/825497188 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3425615822 0 0) 0x7f87b80aba90 con 0x7f87b81430c0 2024-03-25T19:22:24.505 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.506+0000 7f879ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:22:24.505 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.506+0000 7f879ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:24.506 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.506+0000 7f879ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:24.506 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.506+0000 7f879ffff640 10 monclient(hunting): my global_id is 4352 2024-03-25T19:22:24.506 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.506+0000 7f879ffff640 1 -- 172.21.15.83:0/825497188 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87b81daca0 con 0x7f87b81430c0 2024-03-25T19:22:24.515 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.515+0000 7f879ffff640 1 -- 172.21.15.83:0/825497188 <== mon.1 v1:172.21.15.103:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3577966129 0 0) 0x7f87b8138ab0 con 0x7f87b813f710 2024-03-25T19:22:24.516 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.515+0000 7f879ffff640 1 -- 172.21.15.83:0/825497188 >> v1:172.21.15.83:6790/0 conn(0x7f87b81430c0 legacy=0x7f87b81d47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:24.516 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.515+0000 7f879ffff640 1 -- 172.21.15.83:0/825497188 >> v1:172.21.15.83:6789/0 conn(0x7f87b80a98f0 legacy=0x7f87b81383a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:24.516 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.515+0000 7f879ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:22:24.516 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.515+0000 7f879ffff640 1 monclient: found mon.b 2024-03-25T19:22:24.516 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.515+0000 7f879ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:24.516 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.515+0000 7f879ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:22:24.516 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.515+0000 7f879ffff640 1 -- 172.21.15.83:0/825497188 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87b81d41e0 con 0x7f87b813f710 2024-03-25T19:22:24.516 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.515+0000 7f879ffff640 10 monclient: _finish_auth 0 2024-03-25T19:22:24.516 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.515+0000 7f879ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:22:24.516 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.515+0000 7f879ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:24.516 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.516+0000 7f87bdf8c640 5 monclient: authenticate success, global_id 4345 2024-03-25T19:22:24.516 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.516+0000 7f87bdf8c640 10 monclient: _renew_subs 2024-03-25T19:22:24.516 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.516+0000 7f87bdf8c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:22:24.516 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.516+0000 7f87bdf8c640 1 -- 172.21.15.83:0/825497188 --> v1:172.21.15.103:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87b81db840 con 0x7f87b813f710 2024-03-25T19:22:24.517 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.516+0000 7f87bdf8c640 10 client.4345.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:22:24.517 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.516+0000 7f87bdf8c640 10 monclient: _renew_subs 2024-03-25T19:22:24.517 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.516+0000 7f87bdf8c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:22:24.517 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.516+0000 7f87bdf8c640 1 -- 172.21.15.83:0/825497188 --> v1:172.21.15.103:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f87b8138f00 con 0x7f87b813f710 2024-03-25T19:22:24.517 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.516+0000 7f87bdf8c640 1 librados: init done 2024-03-25T19:22:24.517 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.517+0000 7f87bdf8c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:24.517 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.517+0000 7f87bdf8c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:24.517 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.517+0000 7f87bdf8c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:22:24.518 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.517+0000 7f87bdf8c640 1 -- 172.21.15.83:0/825497188 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87b81dae50 con 0x7f87b813f710 2024-03-25T19:22:24.803 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.803+0000 7f879ffff640 1 -- 172.21.15.83:0/825497188 <== mon.1 v1:172.21.15.103:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f87ac001c50 con 0x7f87b813f710 2024-03-25T19:22:24.803 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.803+0000 7f879ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:24.803 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.803+0000 7f879ffff640 1 -- 172.21.15.83:0/825497188 <== mon.1 v1:172.21.15.103:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f87ac0038c0 con 0x7f87b813f710 2024-03-25T19:22:24.803 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.803+0000 7f879ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:24.803 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.803+0000 7f879ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:22:24.803 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.803+0000 7f879ffff640 10 monclient: dump: 2024-03-25T19:22:24.803 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:22:24.803 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:22:24.803 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:24.803 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:24.803 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:24.803 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:22:24.803 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:22:24.803 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:22:24.804 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:22:24.804 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:24.804 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.803+0000 7f87b5ffb640 4 set_mon_vals no callback set 2024-03-25T19:22:24.804 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.804+0000 7f879ffff640 1 -- 172.21.15.83:0/825497188 <== mon.1 v1:172.21.15.103:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7f87ac003ae0 con 0x7f87b813f710 2024-03-25T19:22:24.804 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.805+0000 7f87b77fe640 10 monclient: get_auth_request con 0x7f87a406de10 auth_method 0 2024-03-25T19:22:24.804 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.805+0000 7f879ffff640 10 client.4345.objecter ms_handle_connect 0x7f87a406de10 2024-03-25T19:22:24.812 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.813+0000 7f879ffff640 1 -- 172.21.15.83:0/825497188 <== mon.1 v1:172.21.15.103:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7f87ac0546b0 con 0x7f87b813f710 2024-03-25T19:22:24.812 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.813+0000 7f879ffff640 10 client.4345.objecter ms_dispatch 0x7f87b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:22:24.812 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.813+0000 7f879ffff640 3 client.4345.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:22:24.813 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.813+0000 7f879ffff640 3 client.4345.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:22:24.813 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.813+0000 7f879ffff640 20 client.4345.objecter dump_active .. 0 homeless 2024-03-25T19:22:24.814 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.814+0000 7f879ffff640 1 -- 172.21.15.83:0/825497188 <== mon.1 v1:172.21.15.103:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f87ac0501c0 con 0x7f87b813f710 2024-03-25T19:22:24.814 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.814+0000 7f879ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:24.814 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:24.814+0000 7f879ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:22:25.038 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:25.038+0000 7f87bdf8c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:25.038 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:25.038+0000 7f87bdf8c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:25.038 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:25.038+0000 7f87bdf8c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:22:25.038 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:25.038+0000 7f87bdf8c640 1 -- 172.21.15.83:0/825497188 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f87b80627b0 con 0x7f87b813f710 2024-03-25T19:22:25.116 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:25.116+0000 7f879ffff640 1 -- 172.21.15.83:0/825497188 <== mon.1 v1:172.21.15.103:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7f87ac0520f0 con 0x7f87b813f710 2024-03-25T19:22:25.116 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:25.116+0000 7f879ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:25.116 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:25.116+0000 7f879ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:22:25.116 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:22:25.116 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:22:25.119 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:25.118+0000 7f87bdf8c640 10 librados: watch_flush enter 2024-03-25T19:22:25.119 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:25.119+0000 7f87bdf8c640 10 librados: watch_flush exit 2024-03-25T19:22:25.119 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:25.119+0000 7f87bdf8c640 20 client.4345.objecter shutdown clearing up homeless session... 2024-03-25T19:22:25.119 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:25.119+0000 7f87bdf8c640 10 client.4345.objecter successfully canceled tick 2024-03-25T19:22:25.119 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:25.119+0000 7f87bdf8c640 1 -- 172.21.15.83:0/825497188 >> v1:172.21.15.103:6800/345072327 conn(0x7f87a406de10 legacy=0x7f87a40702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:25.120 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:25.119+0000 7f87bdf8c640 10 monclient: shutdown 2024-03-25T19:22:25.120 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:25.119+0000 7f87bdf8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:25.120 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:25.119+0000 7f87bdf8c640 1 -- 172.21.15.83:0/825497188 >> v1:172.21.15.103:6789/0 conn(0x7f87b813f710 legacy=0x7f87b81db130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:25.120 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:25.119+0000 7f87bdf8c640 1 -- 172.21.15.83:0/825497188 shutdown_connections 2024-03-25T19:22:25.120 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:25.119+0000 7f87bdf8c640 1 -- 172.21.15.83:0/825497188 >> 172.21.15.83:0/825497188 conn(0x7f87b80b7ea0 msgr2=0x7f87b8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:25.120 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:25.119+0000 7f87bdf8c640 1 -- 172.21.15.83:0/825497188 shutdown_connections 2024-03-25T19:22:25.120 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:25.119+0000 7f87bdf8c640 1 -- 172.21.15.83:0/825497188 wait complete. 2024-03-25T19:22:25.120 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:25.119+0000 7f87bdf8c640 1 librados: shutdown 2024-03-25T19:22:25.130 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:22:31.937 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:22:32.051 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.052+0000 7f7b4770e640 10 monclient: get_monmap_and_config 2024-03-25T19:22:32.052 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.052+0000 7f7b4770e640 10 monclient: build_initial_monmap 2024-03-25T19:22:32.052 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.052+0000 7f7b4770e640 10 monclient: monmap: 2024-03-25T19:22:32.052 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:22:32.052 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:22:32.052 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:22:32.053186+0000 2024-03-25T19:22:32.052 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:22:32.053186+0000 2024-03-25T19:22:32.052 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:32.052 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:22:32.052 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:22:32.052 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:22:32.052 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:22:32.052 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:32.053 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.052+0000 7f7b4770e640 1 Processor -- start 2024-03-25T19:22:32.053 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.053+0000 7f7b4770e640 1 -- start start 2024-03-25T19:22:32.053 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.053+0000 7f7b4770e640 10 monclient: init 2024-03-25T19:22:32.054 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.053+0000 7f7b4770e640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:32.054 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.053+0000 7f7b4770e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:22:32.054 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.053+0000 7f7b4770e640 10 monclient(hunting): picked mon.noname-a con 0x7f7b400b10d0 addr v1:172.21.15.83:6789/0 2024-03-25T19:22:32.054 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.053+0000 7f7b4770e640 10 monclient(hunting): picked mon.noname-c con 0x7f7b400b4360 addr v1:172.21.15.103:6789/0 2024-03-25T19:22:32.054 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.053+0000 7f7b4770e640 10 monclient(hunting): picked mon.noname-b con 0x7f7b400a9470 addr v1:172.21.15.83:6790/0 2024-03-25T19:22:32.054 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.054+0000 7f7b4770e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b400a9ab0 con 0x7f7b400b10d0 2024-03-25T19:22:32.054 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.054+0000 7f7b4770e640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b400b2400 con 0x7f7b400b4360 2024-03-25T19:22:32.054 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.054+0000 7f7b4770e640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b400aa2a0 con 0x7f7b400a9470 2024-03-25T19:22:32.054 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.054+0000 7f7b4770e640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:32.054 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.054+0000 7f7b4770e640 10 monclient(hunting): authenticate will time out at 6209.325830s 2024-03-25T19:22:32.062 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.063+0000 7f7b45cab640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f7b400a9470 0x7f7b4014dbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:56312/0 (socket says 172.21.15.83:56312) 2024-03-25T19:22:32.062 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.063+0000 7f7b45cab640 1 -- 172.21.15.83:0/465130309 learned_addr learned my addr 172.21.15.83:0/465130309 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:22:32.078 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.078+0000 7f7b37fff640 1 -- 172.21.15.83:0/465130309 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3779593809 0 0) 0x7f7b400a9ab0 con 0x7f7b400b10d0 2024-03-25T19:22:32.078 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.078+0000 7f7b37fff640 10 monclient(hunting): _negotiate 2024-03-25T19:22:32.078 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.078+0000 7f7b37fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:32.078 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.078+0000 7f7b37fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:32.078 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.078+0000 7f7b37fff640 10 monclient(hunting): my global_id is 4344 2024-03-25T19:22:32.078 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.078+0000 7f7b37fff640 1 -- 172.21.15.83:0/465130309 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b2c003520 con 0x7f7b400b10d0 2024-03-25T19:22:32.079 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.079+0000 7f7b37fff640 1 -- 172.21.15.83:0/465130309 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2440015563 0 0) 0x7f7b400aa2a0 con 0x7f7b400a9470 2024-03-25T19:22:32.079 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.079+0000 7f7b37fff640 10 monclient(hunting): _negotiate 2024-03-25T19:22:32.079 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.079+0000 7f7b37fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:32.079 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.079+0000 7f7b37fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:32.079 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.079+0000 7f7b37fff640 10 monclient(hunting): my global_id is 4355 2024-03-25T19:22:32.079 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.079+0000 7f7b37fff640 1 -- 172.21.15.83:0/465130309 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b400a9ab0 con 0x7f7b400a9470 2024-03-25T19:22:32.081 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.081+0000 7f7b37fff640 1 -- 172.21.15.83:0/465130309 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1877715465 0 0) 0x7f7b400b2400 con 0x7f7b400b4360 2024-03-25T19:22:32.081 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.081+0000 7f7b37fff640 10 monclient(hunting): _negotiate 2024-03-25T19:22:32.081 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.081+0000 7f7b37fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:32.081 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.081+0000 7f7b37fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:32.081 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.081+0000 7f7b37fff640 10 monclient(hunting): my global_id is 4348 2024-03-25T19:22:32.081 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.081+0000 7f7b37fff640 1 -- 172.21.15.83:0/465130309 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b400aa2a0 con 0x7f7b400b4360 2024-03-25T19:22:32.098 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.098+0000 7f7b37fff640 1 -- 172.21.15.83:0/465130309 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2544335603 0 0) 0x7f7b400a9ab0 con 0x7f7b400a9470 2024-03-25T19:22:32.098 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.098+0000 7f7b37fff640 1 -- 172.21.15.83:0/465130309 >> v1:172.21.15.103:6789/0 conn(0x7f7b400b4360 legacy=0x7f7b400b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:32.098 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.098+0000 7f7b37fff640 1 -- 172.21.15.83:0/465130309 >> v1:172.21.15.83:6789/0 conn(0x7f7b400b10d0 legacy=0x7f7b400b4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:32.098 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.098+0000 7f7b37fff640 10 monclient: _finish_hunting 0 2024-03-25T19:22:32.098 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.098+0000 7f7b37fff640 1 monclient: found mon.noname-b 2024-03-25T19:22:32.098 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.098+0000 7f7b37fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:32.098 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.098+0000 7f7b37fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.83:6790/0 2024-03-25T19:22:32.098 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.098+0000 7f7b37fff640 1 -- 172.21.15.83:0/465130309 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b400b14b0 con 0x7f7b400a9470 2024-03-25T19:22:32.098 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.098+0000 7f7b37fff640 10 monclient: _finish_auth 0 2024-03-25T19:22:32.098 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.098+0000 7f7b37fff640 10 monclient: _check_auth_tickets 2024-03-25T19:22:32.098 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.098+0000 7f7b37fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:32.098 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.098+0000 7f7b37fff640 1 -- 172.21.15.83:0/465130309 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f7b380019f0 con 0x7f7b400a9470 2024-03-25T19:22:32.098 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.098+0000 7f7b37fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:32.099 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.098+0000 7f7b37fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:22:32.099 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.098+0000 7f7b37fff640 10 monclient: dump: 2024-03-25T19:22:32.099 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:22:32.099 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:22:32.099 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:32.099 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:32.099 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:32.099 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:22:32.099 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:22:32.099 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:22:32.099 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:22:32.099 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:32.099 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.098+0000 7f7b4770e640 5 monclient: authenticate success, global_id 4355 2024-03-25T19:22:32.099 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.098+0000 7f7b4770e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:32.112 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.112+0000 7f7b37fff640 1 -- 172.21.15.83:0/465130309 <== mon.2 v1:172.21.15.83:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b38001b50 con 0x7f7b400a9470 2024-03-25T19:22:32.112 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.112+0000 7f7b37fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:32.112 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.112+0000 7f7b37fff640 1 -- 172.21.15.83:0/465130309 <== mon.2 v1:172.21.15.83:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f7b38001f60 con 0x7f7b400a9470 2024-03-25T19:22:32.112 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.112+0000 7f7b4770e640 10 monclient: get_monmap_and_config success 2024-03-25T19:22:32.112 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.112+0000 7f7b4770e640 4 set_mon_vals no callback set 2024-03-25T19:22:32.112 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.112+0000 7f7b37fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:32.112 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.112+0000 7f7b37fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:22:32.112 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.112+0000 7f7b37fff640 10 monclient: dump: 2024-03-25T19:22:32.112 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:22:32.112 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:22:32.113 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:32.113 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:32.113 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:32.113 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:22:32.113 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:22:32.113 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:22:32.113 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:22:32.113 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:32.114 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.113+0000 7f7b4770e640 10 monclient: shutdown 2024-03-25T19:22:32.114 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.113+0000 7f7b4770e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:32.114 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.113+0000 7f7b4770e640 1 -- 172.21.15.83:0/465130309 >> v1:172.21.15.83:6790/0 conn(0x7f7b400a9470 legacy=0x7f7b4014dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:32.114 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.113+0000 7f7b4770e640 1 -- 172.21.15.83:0/465130309 shutdown_connections 2024-03-25T19:22:32.114 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.113+0000 7f7b4770e640 1 -- 172.21.15.83:0/465130309 >> 172.21.15.83:0/465130309 conn(0x7f7b400617a0 msgr2=0x7f7b40061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:32.114 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.113+0000 7f7b4770e640 1 -- 172.21.15.83:0/465130309 shutdown_connections 2024-03-25T19:22:32.114 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.113+0000 7f7b4770e640 1 -- 172.21.15.83:0/465130309 wait complete. 2024-03-25T19:22:32.114 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.114+0000 7f7b4770e640 10 monclient: build_initial_monmap 2024-03-25T19:22:32.114 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.114+0000 7f7b4770e640 10 monclient: monmap: 2024-03-25T19:22:32.114 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:22:32.114 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:22:32.114 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:22:32.114 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:22:32.115 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:32.115 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:22:32.115 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:22:32.115 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:22:32.115 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:22:32.115 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:32.116 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.115+0000 7f7b4770e640 1 librados: starting msgr at 2024-03-25T19:22:32.116 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.115+0000 7f7b4770e640 1 librados: starting objecter 2024-03-25T19:22:32.116 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.115+0000 7f7b4770e640 1 Processor -- start 2024-03-25T19:22:32.116 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.115+0000 7f7b4770e640 1 -- start start 2024-03-25T19:22:32.116 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.115+0000 7f7b4770e640 1 librados: setting wanted keys 2024-03-25T19:22:32.117 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.115+0000 7f7b4770e640 1 librados: calling monclient init 2024-03-25T19:22:32.117 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.115+0000 7f7b4770e640 10 monclient: init 2024-03-25T19:22:32.117 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.116+0000 7f7b4770e640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:32.117 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.116+0000 7f7b4770e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:22:32.117 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.116+0000 7f7b4770e640 10 monclient(hunting): picked mon.noname-b con 0x7f7b400a9470 addr v1:172.21.15.103:6789/0 2024-03-25T19:22:32.117 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.116+0000 7f7b4770e640 10 monclient(hunting): picked mon.noname-a con 0x7f7b400b10d0 addr v1:172.21.15.83:6789/0 2024-03-25T19:22:32.117 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.116+0000 7f7b4770e640 10 monclient(hunting): picked mon.noname-c con 0x7f7b400b4360 addr v1:172.21.15.83:6790/0 2024-03-25T19:22:32.117 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.116+0000 7f7b4770e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b400ab930 con 0x7f7b400b10d0 2024-03-25T19:22:32.117 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.116+0000 7f7b4770e640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b400ad8c0 con 0x7f7b400a9470 2024-03-25T19:22:32.117 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.116+0000 7f7b4770e640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b4013a710 con 0x7f7b400b4360 2024-03-25T19:22:32.117 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.116+0000 7f7b4770e640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:32.117 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.116+0000 7f7b4770e640 10 monclient(hunting): authenticate will time out at 6209.388056s 2024-03-25T19:22:32.119 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.120+0000 7f7b45cab640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f7b400b4360 0x7f7b400abdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:56380/0 (socket says 172.21.15.83:56380) 2024-03-25T19:22:32.119 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.120+0000 7f7b45cab640 1 -- 172.21.15.83:0/4225589615 learned_addr learned my addr 172.21.15.83:0/4225589615 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:22:32.120 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.120+0000 7f7b35ffb640 10 client.?.objecter ms_handle_connect 0x7f7b400b4360 2024-03-25T19:22:32.120 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.120+0000 7f7b35ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:32.128 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.128+0000 7f7b35ffb640 10 client.?.objecter ms_handle_connect 0x7f7b400a9470 2024-03-25T19:22:32.128 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.128+0000 7f7b35ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:32.130 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.130+0000 7f7b35ffb640 1 -- 172.21.15.83:0/4225589615 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2973919804 0 0) 0x7f7b4013a710 con 0x7f7b400b4360 2024-03-25T19:22:32.130 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.130+0000 7f7b35ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:22:32.130 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.130+0000 7f7b35ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:32.130 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.130+0000 7f7b35ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:32.130 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.130+0000 7f7b35ffb640 10 monclient(hunting): my global_id is 4358 2024-03-25T19:22:32.130 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.130+0000 7f7b35ffb640 1 -- 172.21.15.83:0/4225589615 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b24004eb0 con 0x7f7b400b4360 2024-03-25T19:22:32.142 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.142+0000 7f7b35ffb640 1 -- 172.21.15.83:0/4225589615 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3267223446 0 0) 0x7f7b400ad8c0 con 0x7f7b400a9470 2024-03-25T19:22:32.142 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.143+0000 7f7b35ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:22:32.142 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.143+0000 7f7b35ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:32.142 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.143+0000 7f7b35ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:32.142 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.143+0000 7f7b35ffb640 10 monclient(hunting): my global_id is 4351 2024-03-25T19:22:32.143 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.143+0000 7f7b35ffb640 1 -- 172.21.15.83:0/4225589615 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b4013a710 con 0x7f7b400a9470 2024-03-25T19:22:32.147 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.147+0000 7f7b35ffb640 10 client.?.objecter ms_handle_connect 0x7f7b400b10d0 2024-03-25T19:22:32.147 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.147+0000 7f7b35ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:32.150 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.150+0000 7f7b35ffb640 1 -- 172.21.15.83:0/4225589615 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3769509162 0 0) 0x7f7b24004eb0 con 0x7f7b400b4360 2024-03-25T19:22:32.150 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.150+0000 7f7b35ffb640 1 -- 172.21.15.83:0/4225589615 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b400ad8c0 con 0x7f7b400b4360 2024-03-25T19:22:32.150 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.150+0000 7f7b35ffb640 1 -- 172.21.15.83:0/4225589615 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f7b38001f60 con 0x7f7b400b4360 2024-03-25T19:22:32.150 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.150+0000 7f7b35ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:32.150 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.150+0000 7f7b35ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:22:32.150 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.150+0000 7f7b35ffb640 10 monclient(hunting): dump: 2024-03-25T19:22:32.150 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:22:32.151 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:22:32.151 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:32.151 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:32.151 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:32.151 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:22:32.151 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:22:32.151 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:22:32.151 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:22:32.151 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:32.151 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.150+0000 7f7b35ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:22:32.156 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.156+0000 7f7b35ffb640 1 -- 172.21.15.83:0/4225589615 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 192122526 0 0) 0x7f7b4013a710 con 0x7f7b400a9470 2024-03-25T19:22:32.156 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.156+0000 7f7b35ffb640 1 -- 172.21.15.83:0/4225589615 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b24004eb0 con 0x7f7b400a9470 2024-03-25T19:22:32.156 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.156+0000 7f7b35ffb640 1 -- 172.21.15.83:0/4225589615 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f7b3c001950 con 0x7f7b400a9470 2024-03-25T19:22:32.156 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.156+0000 7f7b35ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:32.156 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.156+0000 7f7b35ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:22:32.156 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.156+0000 7f7b35ffb640 10 monclient(hunting): dump: 2024-03-25T19:22:32.156 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:22:32.156 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:22:32.156 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:32.156 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:32.156 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:32.156 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:22:32.156 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:22:32.157 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:22:32.157 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:22:32.157 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:32.158 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.158+0000 7f7b35ffb640 1 -- 172.21.15.83:0/4225589615 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1641032968 0 0) 0x7f7b400ab930 con 0x7f7b400b10d0 2024-03-25T19:22:32.158 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.158+0000 7f7b35ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:22:32.158 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.158+0000 7f7b35ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:32.158 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.158+0000 7f7b35ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:32.158 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.158+0000 7f7b35ffb640 10 monclient(hunting): my global_id is 4347 2024-03-25T19:22:32.158 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.158+0000 7f7b35ffb640 1 -- 172.21.15.83:0/4225589615 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b4013a710 con 0x7f7b400b10d0 2024-03-25T19:22:32.163 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.163+0000 7f7b35ffb640 1 -- 172.21.15.83:0/4225589615 <== mon.1 v1:172.21.15.103:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2468168477 0 0) 0x7f7b24004eb0 con 0x7f7b400a9470 2024-03-25T19:22:32.163 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.163+0000 7f7b35ffb640 1 -- 172.21.15.83:0/4225589615 >> v1:172.21.15.83:6790/0 conn(0x7f7b400b4360 legacy=0x7f7b400abdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:32.163 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.163+0000 7f7b35ffb640 1 -- 172.21.15.83:0/4225589615 >> v1:172.21.15.83:6789/0 conn(0x7f7b400b10d0 legacy=0x7f7b401dab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:32.163 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.163+0000 7f7b35ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:22:32.164 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.163+0000 7f7b35ffb640 1 monclient: found mon.b 2024-03-25T19:22:32.164 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.163+0000 7f7b35ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:32.164 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.163+0000 7f7b35ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:22:32.164 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.163+0000 7f7b35ffb640 1 -- 172.21.15.83:0/4225589615 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b4013a8a0 con 0x7f7b400a9470 2024-03-25T19:22:32.164 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.163+0000 7f7b35ffb640 10 monclient: _finish_auth 0 2024-03-25T19:22:32.164 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.163+0000 7f7b35ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:22:32.164 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.163+0000 7f7b35ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:32.164 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.163+0000 7f7b4770e640 5 monclient: authenticate success, global_id 4351 2024-03-25T19:22:32.164 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.163+0000 7f7b4770e640 10 monclient: _renew_subs 2024-03-25T19:22:32.164 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.163+0000 7f7b4770e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:22:32.164 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.163+0000 7f7b4770e640 1 -- 172.21.15.83:0/4225589615 --> v1:172.21.15.103:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b401db2a0 con 0x7f7b400a9470 2024-03-25T19:22:32.164 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.163+0000 7f7b4770e640 10 client.4351.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:22:32.164 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.163+0000 7f7b4770e640 10 monclient: _renew_subs 2024-03-25T19:22:32.164 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.163+0000 7f7b4770e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:22:32.164 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.163+0000 7f7b4770e640 1 -- 172.21.15.83:0/4225589615 --> v1:172.21.15.103:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b400ac4d0 con 0x7f7b400a9470 2024-03-25T19:22:32.164 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.163+0000 7f7b4770e640 1 librados: init done 2024-03-25T19:22:32.165 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.164+0000 7f7b4770e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:32.165 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.164+0000 7f7b4770e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:32.165 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.164+0000 7f7b4770e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:22:32.165 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.164+0000 7f7b4770e640 1 -- 172.21.15.83:0/4225589615 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b401db7d0 con 0x7f7b400a9470 2024-03-25T19:22:32.437 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.437+0000 7f7b35ffb640 1 -- 172.21.15.83:0/4225589615 <== mon.1 v1:172.21.15.103:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b3c001d60 con 0x7f7b400a9470 2024-03-25T19:22:32.437 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.437+0000 7f7b35ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:32.437 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.437+0000 7f7b35ffb640 1 -- 172.21.15.83:0/4225589615 <== mon.1 v1:172.21.15.103:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f7b3c0037b0 con 0x7f7b400a9470 2024-03-25T19:22:32.437 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.437+0000 7f7b35ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:32.437 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.437+0000 7f7b35ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:22:32.437 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.437+0000 7f7b35ffb640 10 monclient: dump: 2024-03-25T19:22:32.437 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:22:32.437 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:22:32.437 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:32.437 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:32.437 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:32.437 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:22:32.438 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:22:32.438 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:22:32.438 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:22:32.438 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:32.438 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.437+0000 7f7b377fe640 4 set_mon_vals no callback set 2024-03-25T19:22:32.438 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.438+0000 7f7b35ffb640 1 -- 172.21.15.83:0/4225589615 <== mon.1 v1:172.21.15.103:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7f7b3c003990 con 0x7f7b400a9470 2024-03-25T19:22:32.439 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.438+0000 7f7b44ca9640 10 monclient: get_auth_request con 0x7f7b2406df30 auth_method 0 2024-03-25T19:22:32.439 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.439+0000 7f7b35ffb640 10 client.4351.objecter ms_handle_connect 0x7f7b2406df30 2024-03-25T19:22:32.441 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.441+0000 7f7b35ffb640 1 -- 172.21.15.83:0/4225589615 <== mon.1 v1:172.21.15.103:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7f7b3c053630 con 0x7f7b400a9470 2024-03-25T19:22:32.441 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.441+0000 7f7b35ffb640 10 client.4351.objecter ms_dispatch 0x7f7b40000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:22:32.441 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.441+0000 7f7b35ffb640 3 client.4351.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:22:32.441 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.441+0000 7f7b35ffb640 3 client.4351.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:22:32.441 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.441+0000 7f7b35ffb640 20 client.4351.objecter dump_active .. 0 homeless 2024-03-25T19:22:32.443 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.443+0000 7f7b35ffb640 1 -- 172.21.15.83:0/4225589615 <== mon.1 v1:172.21.15.103:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7b3c04ef70 con 0x7f7b400a9470 2024-03-25T19:22:32.443 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.443+0000 7f7b35ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:32.443 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.443+0000 7f7b35ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:22:32.692 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.692+0000 7f7b4770e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:32.692 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.692+0000 7f7b4770e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:32.692 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.692+0000 7f7b4770e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:22:32.693 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.692+0000 7f7b4770e640 1 -- 172.21.15.83:0/4225589615 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b400627b0 con 0x7f7b400a9470 2024-03-25T19:22:32.712 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.712+0000 7f7b35ffb640 1 -- 172.21.15.83:0/4225589615 <== mon.1 v1:172.21.15.103:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7f7b3c051070 con 0x7f7b400a9470 2024-03-25T19:22:32.712 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.712+0000 7f7b35ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:32.712 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.712+0000 7f7b35ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:22:32.712 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:22:32.712 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:22:32.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.714+0000 7f7b4770e640 10 librados: watch_flush enter 2024-03-25T19:22:32.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.714+0000 7f7b4770e640 10 librados: watch_flush exit 2024-03-25T19:22:32.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.714+0000 7f7b4770e640 20 client.4351.objecter shutdown clearing up homeless session... 2024-03-25T19:22:32.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.714+0000 7f7b4770e640 10 client.4351.objecter successfully canceled tick 2024-03-25T19:22:32.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.715+0000 7f7b4770e640 1 -- 172.21.15.83:0/4225589615 >> v1:172.21.15.103:6800/345072327 conn(0x7f7b2406df30 legacy=0x7f7b240703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:32.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.715+0000 7f7b4770e640 10 monclient: shutdown 2024-03-25T19:22:32.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.715+0000 7f7b4770e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:32.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.715+0000 7f7b4770e640 1 -- 172.21.15.83:0/4225589615 >> v1:172.21.15.103:6789/0 conn(0x7f7b400a9470 legacy=0x7f7b4013a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:32.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.715+0000 7f7b4770e640 1 -- 172.21.15.83:0/4225589615 shutdown_connections 2024-03-25T19:22:32.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.715+0000 7f7b4770e640 1 -- 172.21.15.83:0/4225589615 >> 172.21.15.83:0/4225589615 conn(0x7f7b400617a0 msgr2=0x7f7b40074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:32.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.715+0000 7f7b4770e640 1 -- 172.21.15.83:0/4225589615 shutdown_connections 2024-03-25T19:22:32.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.715+0000 7f7b4770e640 1 -- 172.21.15.83:0/4225589615 wait complete. 2024-03-25T19:22:32.716 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:32.715+0000 7f7b4770e640 1 librados: shutdown 2024-03-25T19:22:32.725 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:22:39.532 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:22:39.645 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.645+0000 7fb088f9d640 10 monclient: get_monmap_and_config 2024-03-25T19:22:39.645 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.645+0000 7fb088f9d640 10 monclient: build_initial_monmap 2024-03-25T19:22:39.645 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.645+0000 7fb088f9d640 10 monclient: monmap: 2024-03-25T19:22:39.645 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:22:39.645 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:22:39.645 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:22:39.646407+0000 2024-03-25T19:22:39.645 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:22:39.646407+0000 2024-03-25T19:22:39.645 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:39.645 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:22:39.645 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:22:39.645 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:22:39.645 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:22:39.646 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:39.647 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.646+0000 7fb088f9d640 1 Processor -- start 2024-03-25T19:22:39.647 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.646+0000 7fb088f9d640 1 -- start start 2024-03-25T19:22:39.647 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.646+0000 7fb088f9d640 10 monclient: init 2024-03-25T19:22:39.647 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.647+0000 7fb088f9d640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:39.647 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.647+0000 7fb088f9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:22:39.647 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.647+0000 7fb088f9d640 10 monclient(hunting): picked mon.noname-a con 0x7fb0840a98f0 addr v1:172.21.15.83:6789/0 2024-03-25T19:22:39.647 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.647+0000 7fb088f9d640 10 monclient(hunting): picked mon.noname-b con 0x7fb08413f710 addr v1:172.21.15.83:6790/0 2024-03-25T19:22:39.647 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.647+0000 7fb088f9d640 10 monclient(hunting): picked mon.noname-c con 0x7fb0841430c0 addr v1:172.21.15.103:6789/0 2024-03-25T19:22:39.647 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.647+0000 7fb088f9d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb084142520 con 0x7fb0840a98f0 2024-03-25T19:22:39.648 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.647+0000 7fb088f9d640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0840b28a0 con 0x7fb0841430c0 2024-03-25T19:22:39.648 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.647+0000 7fb088f9d640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0840b2a30 con 0x7fb08413f710 2024-03-25T19:22:39.648 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.647+0000 7fb088f9d640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:39.648 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.647+0000 7fb088f9d640 10 monclient(hunting): authenticate will time out at 6216.919223s 2024-03-25T19:22:39.652 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.653+0000 7fb082d9d640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fb0841430c0 0x7fb08414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:46314/0 (socket says 172.21.15.83:46314) 2024-03-25T19:22:39.652 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.653+0000 7fb082d9d640 1 -- 172.21.15.83:0/2057884186 learned_addr learned my addr 172.21.15.83:0/2057884186 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:22:39.662 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.662+0000 7fb08159a640 1 -- 172.21.15.83:0/2057884186 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1179343852 0 0) 0x7fb0840b28a0 con 0x7fb0841430c0 2024-03-25T19:22:39.662 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.662+0000 7fb08159a640 10 monclient(hunting): _negotiate 2024-03-25T19:22:39.662 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.662+0000 7fb08159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:39.662 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.662+0000 7fb08159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:39.662 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.662+0000 7fb08159a640 10 monclient(hunting): my global_id is 4354 2024-03-25T19:22:39.662 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.662+0000 7fb08159a640 1 -- 172.21.15.83:0/2057884186 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb06c003520 con 0x7fb0841430c0 2024-03-25T19:22:39.668 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.667+0000 7fb08159a640 1 -- 172.21.15.83:0/2057884186 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1737586211 0 0) 0x7fb0840b2a30 con 0x7fb08413f710 2024-03-25T19:22:39.668 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.667+0000 7fb08159a640 10 monclient(hunting): _negotiate 2024-03-25T19:22:39.668 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.667+0000 7fb08159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:39.668 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.667+0000 7fb08159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:39.668 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.667+0000 7fb08159a640 10 monclient(hunting): my global_id is 4361 2024-03-25T19:22:39.668 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.667+0000 7fb08159a640 1 -- 172.21.15.83:0/2057884186 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0840b28a0 con 0x7fb08413f710 2024-03-25T19:22:39.676 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.676+0000 7fb08159a640 1 -- 172.21.15.83:0/2057884186 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2680300278 0 0) 0x7fb06c003520 con 0x7fb0841430c0 2024-03-25T19:22:39.677 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.676+0000 7fb08159a640 1 -- 172.21.15.83:0/2057884186 >> v1:172.21.15.83:6790/0 conn(0x7fb08413f710 legacy=0x7fb0841429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:39.677 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.676+0000 7fb08159a640 1 -- 172.21.15.83:0/2057884186 >> v1:172.21.15.83:6789/0 conn(0x7fb0840a98f0 legacy=0x7fb0840a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:39.677 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.676+0000 7fb08159a640 10 monclient: _finish_hunting 0 2024-03-25T19:22:39.677 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.676+0000 7fb08159a640 1 monclient: found mon.noname-c 2024-03-25T19:22:39.677 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.676+0000 7fb08159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:39.677 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.676+0000 7fb08159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.103:6789/0 2024-03-25T19:22:39.677 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.676+0000 7fb08159a640 1 -- 172.21.15.83:0/2057884186 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0840b44c0 con 0x7fb0841430c0 2024-03-25T19:22:39.677 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.676+0000 7fb08159a640 10 monclient: _finish_auth 0 2024-03-25T19:22:39.677 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.676+0000 7fb08159a640 10 monclient: _check_auth_tickets 2024-03-25T19:22:39.677 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.676+0000 7fb08159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:39.677 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.677+0000 7fb088f9d640 5 monclient: authenticate success, global_id 4354 2024-03-25T19:22:39.677 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.677+0000 7fb088f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:39.677 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.677+0000 7fb08159a640 1 -- 172.21.15.83:0/2057884186 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fb07c001a10 con 0x7fb0841430c0 2024-03-25T19:22:39.677 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.677+0000 7fb08159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:39.678 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.677+0000 7fb08159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:22:39.678 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.677+0000 7fb08159a640 10 monclient: dump: 2024-03-25T19:22:39.678 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:22:39.678 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:22:39.678 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:39.678 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:39.678 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:39.678 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:22:39.678 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:22:39.678 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:22:39.678 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:22:39.678 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:39.678 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.677+0000 7fb088f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:39.685 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.685+0000 7fb08159a640 1 -- 172.21.15.83:0/2057884186 <== mon.1 v1:172.21.15.103:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb07c001b70 con 0x7fb0841430c0 2024-03-25T19:22:39.685 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.685+0000 7fb08159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:39.685 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.685+0000 7fb08159a640 1 -- 172.21.15.83:0/2057884186 <== mon.1 v1:172.21.15.103:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fb07c001f80 con 0x7fb0841430c0 2024-03-25T19:22:39.685 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.685+0000 7fb08159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:39.685 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.685+0000 7fb08159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:22:39.685 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.685+0000 7fb08159a640 10 monclient: dump: 2024-03-25T19:22:39.686 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:22:39.686 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:22:39.686 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:39.686 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:39.686 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:39.686 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:22:39.686 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:22:39.686 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:22:39.686 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:22:39.686 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:39.686 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.685+0000 7fb088f9d640 10 monclient: get_monmap_and_config success 2024-03-25T19:22:39.686 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.685+0000 7fb088f9d640 4 set_mon_vals no callback set 2024-03-25T19:22:39.688 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.686+0000 7fb088f9d640 10 monclient: shutdown 2024-03-25T19:22:39.688 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.686+0000 7fb088f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:39.688 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.686+0000 7fb088f9d640 1 -- 172.21.15.83:0/2057884186 >> v1:172.21.15.103:6789/0 conn(0x7fb0841430c0 legacy=0x7fb08414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:39.688 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.686+0000 7fb088f9d640 1 -- 172.21.15.83:0/2057884186 shutdown_connections 2024-03-25T19:22:39.689 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.686+0000 7fb088f9d640 1 -- 172.21.15.83:0/2057884186 >> 172.21.15.83:0/2057884186 conn(0x7fb0840b7ea0 msgr2=0x7fb0840b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:39.689 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.686+0000 7fb088f9d640 1 -- 172.21.15.83:0/2057884186 shutdown_connections 2024-03-25T19:22:39.689 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.686+0000 7fb088f9d640 1 -- 172.21.15.83:0/2057884186 wait complete. 2024-03-25T19:22:39.689 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.687+0000 7fb088f9d640 10 monclient: build_initial_monmap 2024-03-25T19:22:39.689 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.687+0000 7fb088f9d640 10 monclient: monmap: 2024-03-25T19:22:39.689 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:22:39.689 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:22:39.689 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:22:39.689 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:22:39.689 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:39.689 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:22:39.689 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:22:39.689 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:22:39.689 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:22:39.689 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:39.690 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.687+0000 7fb088f9d640 1 librados: starting msgr at 2024-03-25T19:22:39.690 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.687+0000 7fb088f9d640 1 librados: starting objecter 2024-03-25T19:22:39.690 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.688+0000 7fb088f9d640 1 Processor -- start 2024-03-25T19:22:39.690 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.688+0000 7fb088f9d640 1 -- start start 2024-03-25T19:22:39.690 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.688+0000 7fb088f9d640 1 librados: setting wanted keys 2024-03-25T19:22:39.690 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.688+0000 7fb088f9d640 1 librados: calling monclient init 2024-03-25T19:22:39.690 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.688+0000 7fb088f9d640 10 monclient: init 2024-03-25T19:22:39.690 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.688+0000 7fb088f9d640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:39.690 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.688+0000 7fb088f9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:22:39.690 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.688+0000 7fb088f9d640 10 monclient(hunting): picked mon.noname-c con 0x7fb0840a98f0 addr v1:172.21.15.83:6790/0 2024-03-25T19:22:39.690 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.688+0000 7fb088f9d640 10 monclient(hunting): picked mon.noname-a con 0x7fb08413f710 addr v1:172.21.15.83:6789/0 2024-03-25T19:22:39.690 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.688+0000 7fb088f9d640 10 monclient(hunting): picked mon.noname-b con 0x7fb0841430c0 addr v1:172.21.15.103:6789/0 2024-03-25T19:22:39.690 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.688+0000 7fb088f9d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0841354c0 con 0x7fb08413f710 2024-03-25T19:22:39.690 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.688+0000 7fb088f9d640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0841319b0 con 0x7fb0841430c0 2024-03-25T19:22:39.691 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.688+0000 7fb088f9d640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb084135070 con 0x7fb0840a98f0 2024-03-25T19:22:39.691 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.688+0000 7fb088f9d640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:39.691 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.688+0000 7fb088f9d640 10 monclient(hunting): authenticate will time out at 6216.960383s 2024-03-25T19:22:39.691 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.691+0000 7fb08259c640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fb0840a98f0 0x7fb0841d7500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:51174/0 (socket says 172.21.15.83:51174) 2024-03-25T19:22:39.691 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.691+0000 7fb08259c640 1 -- 172.21.15.83:0/3917266110 learned_addr learned my addr 172.21.15.83:0/3917266110 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:22:39.695 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.695+0000 7fb072ffd640 10 client.?.objecter ms_handle_connect 0x7fb0840a98f0 2024-03-25T19:22:39.695 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.695+0000 7fb072ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:39.707 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.707+0000 7fb072ffd640 1 -- 172.21.15.83:0/3917266110 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3375743348 0 0) 0x7fb084135070 con 0x7fb0840a98f0 2024-03-25T19:22:39.707 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.707+0000 7fb072ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:22:39.707 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.707+0000 7fb072ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:39.707 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.707+0000 7fb072ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:39.707 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.707+0000 7fb072ffd640 10 monclient(hunting): my global_id is 4364 2024-03-25T19:22:39.708 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.708+0000 7fb072ffd640 1 -- 172.21.15.83:0/3917266110 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb064004eb0 con 0x7fb0840a98f0 2024-03-25T19:22:39.708 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.708+0000 7fb072ffd640 10 client.?.objecter ms_handle_connect 0x7fb08413f710 2024-03-25T19:22:39.708 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.708+0000 7fb072ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:39.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.715+0000 7fb072ffd640 10 client.?.objecter ms_handle_connect 0x7fb0841430c0 2024-03-25T19:22:39.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.715+0000 7fb072ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:39.723 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.723+0000 7fb072ffd640 1 -- 172.21.15.83:0/3917266110 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1442650258 0 0) 0x7fb0841354c0 con 0x7fb08413f710 2024-03-25T19:22:39.723 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.723+0000 7fb072ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:22:39.723 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.723+0000 7fb072ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:39.723 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.723+0000 7fb072ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:39.723 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.723+0000 7fb072ffd640 10 monclient(hunting): my global_id is 4353 2024-03-25T19:22:39.723 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.723+0000 7fb072ffd640 1 -- 172.21.15.83:0/3917266110 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb084135070 con 0x7fb08413f710 2024-03-25T19:22:39.730 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.730+0000 7fb072ffd640 1 -- 172.21.15.83:0/3917266110 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2093407631 0 0) 0x7fb064004eb0 con 0x7fb0840a98f0 2024-03-25T19:22:39.730 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.730+0000 7fb072ffd640 1 -- 172.21.15.83:0/3917266110 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0841354c0 con 0x7fb0840a98f0 2024-03-25T19:22:39.730 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.730+0000 7fb072ffd640 1 -- 172.21.15.83:0/3917266110 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fb074001a30 con 0x7fb0840a98f0 2024-03-25T19:22:39.730 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.730+0000 7fb072ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:39.730 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.730+0000 7fb072ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:22:39.730 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.730+0000 7fb072ffd640 10 monclient(hunting): dump: 2024-03-25T19:22:39.730 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:22:39.730 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:22:39.730 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:39.731 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:39.731 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:39.731 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:22:39.731 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:22:39.731 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:22:39.731 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:22:39.731 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:39.731 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.730+0000 7fb072ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:22:39.732 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.732+0000 7fb072ffd640 1 -- 172.21.15.83:0/3917266110 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 445910006 0 0) 0x7fb0841319b0 con 0x7fb0841430c0 2024-03-25T19:22:39.732 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.732+0000 7fb072ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:22:39.732 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.732+0000 7fb072ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:39.732 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.732+0000 7fb072ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:39.732 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.732+0000 7fb072ffd640 10 monclient(hunting): my global_id is 4357 2024-03-25T19:22:39.732 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.732+0000 7fb072ffd640 1 -- 172.21.15.83:0/3917266110 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb064004eb0 con 0x7fb0841430c0 2024-03-25T19:22:39.736 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.736+0000 7fb072ffd640 1 -- 172.21.15.83:0/3917266110 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2403490275 0 0) 0x7fb0841354c0 con 0x7fb0840a98f0 2024-03-25T19:22:39.736 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.736+0000 7fb072ffd640 1 -- 172.21.15.83:0/3917266110 >> v1:172.21.15.103:6789/0 conn(0x7fb0841430c0 legacy=0x7fb0841312a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:39.736 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.736+0000 7fb072ffd640 1 -- 172.21.15.83:0/3917266110 >> v1:172.21.15.83:6789/0 conn(0x7fb08413f710 legacy=0x7fb084130b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:39.736 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.736+0000 7fb072ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:22:39.736 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.736+0000 7fb072ffd640 1 monclient: found mon.c 2024-03-25T19:22:39.736 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.736+0000 7fb072ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:39.736 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.736+0000 7fb072ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:22:39.736 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.736+0000 7fb072ffd640 1 -- 172.21.15.83:0/3917266110 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb084135200 con 0x7fb0840a98f0 2024-03-25T19:22:39.737 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.736+0000 7fb072ffd640 10 monclient: _finish_auth 0 2024-03-25T19:22:39.737 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.736+0000 7fb072ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:22:39.737 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.736+0000 7fb072ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:39.737 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.736+0000 7fb088f9d640 5 monclient: authenticate success, global_id 4364 2024-03-25T19:22:39.737 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.736+0000 7fb088f9d640 10 monclient: _renew_subs 2024-03-25T19:22:39.737 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.736+0000 7fb088f9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:22:39.737 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.736+0000 7fb088f9d640 1 -- 172.21.15.83:0/3917266110 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0840b7010 con 0x7fb0840a98f0 2024-03-25T19:22:39.737 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.736+0000 7fb088f9d640 10 client.4364.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:22:39.737 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.736+0000 7fb088f9d640 10 monclient: _renew_subs 2024-03-25T19:22:39.737 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.736+0000 7fb088f9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:22:39.737 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.736+0000 7fb088f9d640 1 -- 172.21.15.83:0/3917266110 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb08413ba70 con 0x7fb0840a98f0 2024-03-25T19:22:39.737 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.736+0000 7fb088f9d640 1 librados: init done 2024-03-25T19:22:39.738 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.737+0000 7fb088f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:39.738 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.737+0000 7fb088f9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:39.738 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.737+0000 7fb088f9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:22:39.738 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:39.737+0000 7fb088f9d640 1 -- 172.21.15.83:0/3917266110 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0840b73b0 con 0x7fb0840a98f0 2024-03-25T19:22:40.053 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:40.053+0000 7fb072ffd640 1 -- 172.21.15.83:0/3917266110 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb074001b90 con 0x7fb0840a98f0 2024-03-25T19:22:40.053 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:40.053+0000 7fb072ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:40.053 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:40.053+0000 7fb072ffd640 1 -- 172.21.15.83:0/3917266110 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fb074003a60 con 0x7fb0840a98f0 2024-03-25T19:22:40.053 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:40.053+0000 7fb072ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:40.053 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:40.053+0000 7fb072ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:22:40.053 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:40.053+0000 7fb072ffd640 10 monclient: dump: 2024-03-25T19:22:40.053 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:22:40.053 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:22:40.053 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:40.053 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:40.054 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:40.054 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:22:40.054 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:22:40.054 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:22:40.054 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:22:40.054 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:40.054 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:40.053+0000 7fb080d99640 4 set_mon_vals no callback set 2024-03-25T19:22:40.055 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:40.054+0000 7fb072ffd640 1 -- 172.21.15.83:0/3917266110 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7fb074003c80 con 0x7fb0840a98f0 2024-03-25T19:22:40.055 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:40.054+0000 7fb081d9b640 10 monclient: get_auth_request con 0x7fb06406de00 auth_method 0 2024-03-25T19:22:40.055 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:40.054+0000 7fb072ffd640 10 client.4364.objecter ms_handle_connect 0x7fb06406de00 2024-03-25T19:22:40.059 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:40.059+0000 7fb072ffd640 1 -- 172.21.15.83:0/3917266110 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7fb0740548b0 con 0x7fb0840a98f0 2024-03-25T19:22:40.059 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:40.059+0000 7fb072ffd640 10 client.4364.objecter ms_dispatch 0x7fb084000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:22:40.059 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:40.059+0000 7fb072ffd640 3 client.4364.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:22:40.059 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:40.059+0000 7fb072ffd640 3 client.4364.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:22:40.059 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:40.059+0000 7fb072ffd640 20 client.4364.objecter dump_active .. 0 homeless 2024-03-25T19:22:40.059 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:40.059+0000 7fb072ffd640 1 -- 172.21.15.83:0/3917266110 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb0740503c0 con 0x7fb0840a98f0 2024-03-25T19:22:40.060 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:40.060+0000 7fb072ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:40.060 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:40.060+0000 7fb072ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:22:40.305 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:40.306+0000 7fb088f9d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:40.306 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:40.306+0000 7fb088f9d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:40.306 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:40.306+0000 7fb088f9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:22:40.306 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:40.306+0000 7fb088f9d640 1 -- 172.21.15.83:0/3917266110 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb084137160 con 0x7fb0840a98f0 2024-03-25T19:22:40.338 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:40.338+0000 7fb072ffd640 1 -- 172.21.15.83:0/3917266110 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7fb0740522f0 con 0x7fb0840a98f0 2024-03-25T19:22:40.338 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:40.338+0000 7fb072ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:40.338 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:40.338+0000 7fb072ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:22:40.338 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:22:40.338 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:22:40.341 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:40.340+0000 7fb088f9d640 10 librados: watch_flush enter 2024-03-25T19:22:40.341 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:40.341+0000 7fb088f9d640 10 librados: watch_flush exit 2024-03-25T19:22:40.341 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:40.341+0000 7fb088f9d640 20 client.4364.objecter shutdown clearing up homeless session... 2024-03-25T19:22:40.341 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:40.341+0000 7fb088f9d640 10 client.4364.objecter successfully canceled tick 2024-03-25T19:22:40.341 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:40.341+0000 7fb088f9d640 1 -- 172.21.15.83:0/3917266110 >> v1:172.21.15.103:6800/345072327 conn(0x7fb06406de00 legacy=0x7fb0640702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:40.341 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:40.341+0000 7fb088f9d640 10 monclient: shutdown 2024-03-25T19:22:40.341 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:40.341+0000 7fb088f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:40.341 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:40.341+0000 7fb088f9d640 1 -- 172.21.15.83:0/3917266110 >> v1:172.21.15.83:6790/0 conn(0x7fb0840a98f0 legacy=0x7fb0841d7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:40.341 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:40.341+0000 7fb088f9d640 1 -- 172.21.15.83:0/3917266110 shutdown_connections 2024-03-25T19:22:40.341 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:40.341+0000 7fb088f9d640 1 -- 172.21.15.83:0/3917266110 >> 172.21.15.83:0/3917266110 conn(0x7fb0840b7ea0 msgr2=0x7fb08413d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:40.342 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:40.341+0000 7fb088f9d640 1 -- 172.21.15.83:0/3917266110 shutdown_connections 2024-03-25T19:22:40.342 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:40.341+0000 7fb088f9d640 1 -- 172.21.15.83:0/3917266110 wait complete. 2024-03-25T19:22:40.342 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:40.341+0000 7fb088f9d640 1 librados: shutdown 2024-03-25T19:22:40.352 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:22:42.578 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:03:24.505 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:42.625 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:03:24.553 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:42.668 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:03:24.596 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:42.693 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:03:24.604 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:42.697 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:03:24.625 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:42.699 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:03:24.610 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:42.707 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:03:24.634 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:42.719 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:03:24.629 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:42.731 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:03:24.658 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:47.159 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:22:47.276 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.277+0000 7f930e554640 10 monclient: get_monmap_and_config 2024-03-25T19:22:47.276 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.277+0000 7f930e554640 10 monclient: build_initial_monmap 2024-03-25T19:22:47.277 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.277+0000 7f930e554640 10 monclient: monmap: 2024-03-25T19:22:47.277 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:22:47.277 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:22:47.277 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:22:47.278062+0000 2024-03-25T19:22:47.277 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:22:47.278062+0000 2024-03-25T19:22:47.277 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:47.277 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:22:47.277 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:22:47.277 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:22:47.277 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:22:47.277 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:47.278 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.277+0000 7f930e554640 1 Processor -- start 2024-03-25T19:22:47.278 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.278+0000 7f930e554640 1 -- start start 2024-03-25T19:22:47.278 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.278+0000 7f930e554640 10 monclient: init 2024-03-25T19:22:47.278 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.278+0000 7f930e554640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:47.279 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.278+0000 7f930e554640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:22:47.279 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.278+0000 7f930e554640 10 monclient(hunting): picked mon.noname-b con 0x7f93080a98f0 addr v1:172.21.15.83:6790/0 2024-03-25T19:22:47.279 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.278+0000 7f930e554640 10 monclient(hunting): picked mon.noname-c con 0x7f930813f710 addr v1:172.21.15.103:6789/0 2024-03-25T19:22:47.279 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.278+0000 7f930e554640 10 monclient(hunting): picked mon.noname-a con 0x7f93081430c0 addr v1:172.21.15.83:6789/0 2024-03-25T19:22:47.279 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.278+0000 7f930e554640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9308142520 con 0x7f93081430c0 2024-03-25T19:22:47.279 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.278+0000 7f930e554640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93080b28a0 con 0x7f930813f710 2024-03-25T19:22:47.279 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.278+0000 7f930e554640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93080b2a30 con 0x7f93080a98f0 2024-03-25T19:22:47.279 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.278+0000 7f930e554640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:47.279 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.278+0000 7f930e554640 10 monclient(hunting): authenticate will time out at 6224.550464s 2024-03-25T19:22:47.285 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.286+0000 7f9307fff640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f93080a98f0 0x7f93080a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:47414/0 (socket says 172.21.15.83:47414) 2024-03-25T19:22:47.285 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.286+0000 7f9307fff640 1 -- 172.21.15.83:0/3704112585 learned_addr learned my addr 172.21.15.83:0/3704112585 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:22:47.301 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.301+0000 7f9306ffd640 1 -- 172.21.15.83:0/3704112585 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 113331994 0 0) 0x7f93080b2a30 con 0x7f93080a98f0 2024-03-25T19:22:47.301 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.301+0000 7f9306ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:22:47.301 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.301+0000 7f9306ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:47.301 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.301+0000 7f9306ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:47.301 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.301+0000 7f9306ffd640 10 monclient(hunting): my global_id is 4367 2024-03-25T19:22:47.301 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.301+0000 7f9306ffd640 1 -- 172.21.15.83:0/3704112585 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92f0003520 con 0x7f93080a98f0 2024-03-25T19:22:47.304 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.304+0000 7f9306ffd640 1 -- 172.21.15.83:0/3704112585 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3632786845 0 0) 0x7f93080b28a0 con 0x7f930813f710 2024-03-25T19:22:47.304 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.304+0000 7f9306ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:22:47.304 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.304+0000 7f9306ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:47.304 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.304+0000 7f9306ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:47.304 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.304+0000 7f9306ffd640 10 monclient(hunting): my global_id is 4360 2024-03-25T19:22:47.304 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.304+0000 7f9306ffd640 1 -- 172.21.15.83:0/3704112585 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93080b2a30 con 0x7f930813f710 2024-03-25T19:22:47.318 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.318+0000 7f9306ffd640 1 -- 172.21.15.83:0/3704112585 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4275026848 0 0) 0x7f9308142520 con 0x7f93081430c0 2024-03-25T19:22:47.318 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.318+0000 7f9306ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:22:47.318 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.318+0000 7f9306ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:47.318 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.318+0000 7f9306ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:47.318 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.318+0000 7f9306ffd640 10 monclient(hunting): my global_id is 4356 2024-03-25T19:22:47.318 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.318+0000 7f9306ffd640 1 -- 172.21.15.83:0/3704112585 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93080b28a0 con 0x7f93081430c0 2024-03-25T19:22:47.319 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.319+0000 7f9306ffd640 1 -- 172.21.15.83:0/3704112585 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2518838239 0 0) 0x7f92f0003520 con 0x7f93080a98f0 2024-03-25T19:22:47.319 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.319+0000 7f9306ffd640 1 -- 172.21.15.83:0/3704112585 >> v1:172.21.15.103:6789/0 conn(0x7f930813f710 legacy=0x7f93081429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:47.319 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.319+0000 7f9306ffd640 1 -- 172.21.15.83:0/3704112585 >> v1:172.21.15.83:6789/0 conn(0x7f93081430c0 legacy=0x7f930814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:47.320 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.319+0000 7f9306ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:22:47.320 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.319+0000 7f9306ffd640 1 monclient: found mon.noname-b 2024-03-25T19:22:47.320 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.319+0000 7f9306ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:47.320 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.319+0000 7f9306ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.83:6790/0 2024-03-25T19:22:47.320 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.319+0000 7f9306ffd640 1 -- 172.21.15.83:0/3704112585 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93080b44c0 con 0x7f93080a98f0 2024-03-25T19:22:47.320 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.319+0000 7f9306ffd640 10 monclient: _finish_auth 0 2024-03-25T19:22:47.320 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.319+0000 7f9306ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:22:47.320 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.319+0000 7f9306ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:47.320 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.319+0000 7f9306ffd640 1 -- 172.21.15.83:0/3704112585 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f92f8001690 con 0x7f93080a98f0 2024-03-25T19:22:47.320 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.319+0000 7f9306ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:47.320 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.319+0000 7f9306ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:22:47.320 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.319+0000 7f9306ffd640 10 monclient: dump: 2024-03-25T19:22:47.320 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:22:47.320 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:22:47.320 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:47.320 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:47.321 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:47.321 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:22:47.321 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:22:47.321 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:22:47.321 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:22:47.321 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:47.321 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.319+0000 7f930e554640 5 monclient: authenticate success, global_id 4367 2024-03-25T19:22:47.321 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.319+0000 7f930e554640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:47.327 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.328+0000 7f9306ffd640 1 -- 172.21.15.83:0/3704112585 <== mon.2 v1:172.21.15.83:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92f80017f0 con 0x7f93080a98f0 2024-03-25T19:22:47.328 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.328+0000 7f9306ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:47.328 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.328+0000 7f9306ffd640 1 -- 172.21.15.83:0/3704112585 <== mon.2 v1:172.21.15.83:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f92f8001c00 con 0x7f93080a98f0 2024-03-25T19:22:47.328 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.328+0000 7f9306ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:47.328 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.328+0000 7f9306ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:22:47.328 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.328+0000 7f9306ffd640 10 monclient: dump: 2024-03-25T19:22:47.328 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:22:47.328 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:22:47.328 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:47.328 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:47.328 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:47.328 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:22:47.328 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:22:47.328 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:22:47.328 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:22:47.328 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:47.329 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.328+0000 7f930e554640 10 monclient: get_monmap_and_config success 2024-03-25T19:22:47.329 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.328+0000 7f930e554640 4 set_mon_vals no callback set 2024-03-25T19:22:47.331 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.328+0000 7f930e554640 10 monclient: shutdown 2024-03-25T19:22:47.331 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.328+0000 7f930e554640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:47.331 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.328+0000 7f930e554640 1 -- 172.21.15.83:0/3704112585 >> v1:172.21.15.83:6790/0 conn(0x7f93080a98f0 legacy=0x7f93080a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:47.331 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.329+0000 7f930e554640 1 -- 172.21.15.83:0/3704112585 shutdown_connections 2024-03-25T19:22:47.331 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.329+0000 7f930e554640 1 -- 172.21.15.83:0/3704112585 >> 172.21.15.83:0/3704112585 conn(0x7f93080b7ea0 msgr2=0x7f93080b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:47.331 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.329+0000 7f930e554640 1 -- 172.21.15.83:0/3704112585 shutdown_connections 2024-03-25T19:22:47.331 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.329+0000 7f930e554640 1 -- 172.21.15.83:0/3704112585 wait complete. 2024-03-25T19:22:47.331 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.329+0000 7f930e554640 10 monclient: build_initial_monmap 2024-03-25T19:22:47.331 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.329+0000 7f930e554640 10 monclient: monmap: 2024-03-25T19:22:47.332 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:22:47.332 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:22:47.332 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:22:47.332 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:22:47.332 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:47.332 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:22:47.332 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:22:47.332 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:22:47.332 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:22:47.332 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:47.332 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.330+0000 7f930e554640 1 librados: starting msgr at 2024-03-25T19:22:47.332 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.330+0000 7f930e554640 1 librados: starting objecter 2024-03-25T19:22:47.332 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.330+0000 7f930e554640 1 Processor -- start 2024-03-25T19:22:47.332 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.330+0000 7f930e554640 1 -- start start 2024-03-25T19:22:47.332 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.330+0000 7f930e554640 1 librados: setting wanted keys 2024-03-25T19:22:47.332 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.330+0000 7f930e554640 1 librados: calling monclient init 2024-03-25T19:22:47.333 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.330+0000 7f930e554640 10 monclient: init 2024-03-25T19:22:47.333 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.330+0000 7f930e554640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:47.333 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.331+0000 7f930e554640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:22:47.333 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.331+0000 7f930e554640 10 monclient(hunting): picked mon.noname-a con 0x7f93080a98f0 addr v1:172.21.15.83:6789/0 2024-03-25T19:22:47.333 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.331+0000 7f930e554640 10 monclient(hunting): picked mon.noname-b con 0x7f930813f710 addr v1:172.21.15.103:6789/0 2024-03-25T19:22:47.333 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.331+0000 7f930e554640 10 monclient(hunting): picked mon.noname-c con 0x7f93081430c0 addr v1:172.21.15.83:6790/0 2024-03-25T19:22:47.333 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.331+0000 7f930e554640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93081370a0 con 0x7f93080a98f0 2024-03-25T19:22:47.333 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.331+0000 7f930e554640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9308135090 con 0x7f930813f710 2024-03-25T19:22:47.333 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.331+0000 7f930e554640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9308135220 con 0x7f93081430c0 2024-03-25T19:22:47.333 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.331+0000 7f930e554640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:47.333 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.331+0000 7f930e554640 10 monclient(hunting): authenticate will time out at 6224.602977s 2024-03-25T19:22:47.336 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.336+0000 7f930caf1640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f93081430c0 0x7f93081d48f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:47448/0 (socket says 172.21.15.83:47448) 2024-03-25T19:22:47.336 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.336+0000 7f930caf1640 1 -- 172.21.15.83:0/2508987900 learned_addr learned my addr 172.21.15.83:0/2508987900 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:22:47.337 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.337+0000 7f9304ff9640 10 client.?.objecter ms_handle_connect 0x7f93081430c0 2024-03-25T19:22:47.337 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.337+0000 7f9304ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:47.342 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.342+0000 7f9304ff9640 10 client.?.objecter ms_handle_connect 0x7f93080a98f0 2024-03-25T19:22:47.342 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.342+0000 7f9304ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:47.347 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.347+0000 7f9304ff9640 1 -- 172.21.15.83:0/2508987900 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3893109506 0 0) 0x7f9308135220 con 0x7f93081430c0 2024-03-25T19:22:47.347 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.347+0000 7f9304ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:22:47.347 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.347+0000 7f9304ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:47.347 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.347+0000 7f9304ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:47.347 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.347+0000 7f9304ff9640 10 monclient(hunting): my global_id is 4370 2024-03-25T19:22:47.347 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.347+0000 7f9304ff9640 1 -- 172.21.15.83:0/2508987900 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92f4004eb0 con 0x7f93081430c0 2024-03-25T19:22:47.353 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.354+0000 7f9304ff9640 10 client.?.objecter ms_handle_connect 0x7f930813f710 2024-03-25T19:22:47.354 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.354+0000 7f9304ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:47.362 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.362+0000 7f9304ff9640 1 -- 172.21.15.83:0/2508987900 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1761801910 0 0) 0x7f9308135090 con 0x7f930813f710 2024-03-25T19:22:47.362 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.362+0000 7f9304ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:22:47.362 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.362+0000 7f9304ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:47.362 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.362+0000 7f9304ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:47.362 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.362+0000 7f9304ff9640 10 monclient(hunting): my global_id is 4363 2024-03-25T19:22:47.362 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.362+0000 7f9304ff9640 1 -- 172.21.15.83:0/2508987900 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9308135220 con 0x7f930813f710 2024-03-25T19:22:47.366 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.367+0000 7f9304ff9640 1 -- 172.21.15.83:0/2508987900 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3280921982 0 0) 0x7f93081370a0 con 0x7f93080a98f0 2024-03-25T19:22:47.366 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.367+0000 7f9304ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:22:47.366 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.367+0000 7f9304ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:47.367 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.367+0000 7f9304ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:47.367 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.367+0000 7f9304ff9640 10 monclient(hunting): my global_id is 4359 2024-03-25T19:22:47.367 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.367+0000 7f9304ff9640 1 -- 172.21.15.83:0/2508987900 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9308135090 con 0x7f93080a98f0 2024-03-25T19:22:47.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.371+0000 7f9304ff9640 1 -- 172.21.15.83:0/2508987900 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 589484115 0 0) 0x7f92f4004eb0 con 0x7f93081430c0 2024-03-25T19:22:47.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.371+0000 7f9304ff9640 1 -- 172.21.15.83:0/2508987900 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f93081370a0 con 0x7f93081430c0 2024-03-25T19:22:47.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.371+0000 7f9304ff9640 1 -- 172.21.15.83:0/2508987900 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f9300001c70 con 0x7f93081430c0 2024-03-25T19:22:47.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.371+0000 7f9304ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:47.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.371+0000 7f9304ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:22:47.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.371+0000 7f9304ff9640 10 monclient(hunting): dump: 2024-03-25T19:22:47.371 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:22:47.371 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:22:47.371 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:47.371 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:47.371 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:47.371 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:22:47.372 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:22:47.372 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:22:47.372 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:22:47.372 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:47.372 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.371+0000 7f9304ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:22:47.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.383+0000 7f9304ff9640 1 -- 172.21.15.83:0/2508987900 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 632406749 0 0) 0x7f93081370a0 con 0x7f93081430c0 2024-03-25T19:22:47.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.383+0000 7f9304ff9640 1 -- 172.21.15.83:0/2508987900 >> v1:172.21.15.103:6789/0 conn(0x7f930813f710 legacy=0x7f93081d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:47.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.383+0000 7f9304ff9640 1 -- 172.21.15.83:0/2508987900 >> v1:172.21.15.83:6789/0 conn(0x7f93080a98f0 legacy=0x7f93081dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:47.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.383+0000 7f9304ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:22:47.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.383+0000 7f9304ff9640 1 monclient: found mon.c 2024-03-25T19:22:47.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.383+0000 7f9304ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:47.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.383+0000 7f9304ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:22:47.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.383+0000 7f9304ff9640 1 -- 172.21.15.83:0/2508987900 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93081d5000 con 0x7f93081430c0 2024-03-25T19:22:47.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.383+0000 7f9304ff9640 10 monclient: _finish_auth 0 2024-03-25T19:22:47.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.383+0000 7f9304ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:22:47.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.383+0000 7f9304ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:47.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.383+0000 7f930e554640 5 monclient: authenticate success, global_id 4370 2024-03-25T19:22:47.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.383+0000 7f930e554640 10 monclient: _renew_subs 2024-03-25T19:22:47.384 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.383+0000 7f930e554640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:22:47.384 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.383+0000 7f930e554640 1 -- 172.21.15.83:0/2508987900 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93081d52b0 con 0x7f93081430c0 2024-03-25T19:22:47.384 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.383+0000 7f930e554640 10 client.4370.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:22:47.384 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.383+0000 7f930e554640 10 monclient: _renew_subs 2024-03-25T19:22:47.384 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.383+0000 7f930e554640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:22:47.384 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.383+0000 7f930e554640 1 -- 172.21.15.83:0/2508987900 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9308135910 con 0x7f93081430c0 2024-03-25T19:22:47.384 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.383+0000 7f930e554640 1 librados: init done 2024-03-25T19:22:47.384 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.384+0000 7f930e554640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:47.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.384+0000 7f930e554640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:47.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.384+0000 7f930e554640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:22:47.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.384+0000 7f930e554640 1 -- 172.21.15.83:0/2508987900 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93080627b0 con 0x7f93081430c0 2024-03-25T19:22:47.712 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.712+0000 7f9304ff9640 1 -- 172.21.15.83:0/2508987900 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9300003970 con 0x7f93081430c0 2024-03-25T19:22:47.712 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.712+0000 7f9304ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:47.712 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.712+0000 7f9304ff9640 1 -- 172.21.15.83:0/2508987900 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f9300003d80 con 0x7f93081430c0 2024-03-25T19:22:47.712 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.712+0000 7f9304ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:47.712 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.712+0000 7f9304ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:22:47.712 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.712+0000 7f9304ff9640 10 monclient: dump: 2024-03-25T19:22:47.712 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:22:47.712 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:22:47.713 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:47.713 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:47.713 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:47.713 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:22:47.713 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:22:47.713 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:22:47.713 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:22:47.713 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:47.713 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.712+0000 7f93067fc640 4 set_mon_vals no callback set 2024-03-25T19:22:47.713 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.713+0000 7f9304ff9640 1 -- 172.21.15.83:0/2508987900 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7f9300003fa0 con 0x7f93081430c0 2024-03-25T19:22:47.714 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.713+0000 7f9307fff640 10 monclient: get_auth_request con 0x7f92f406ddf0 auth_method 0 2024-03-25T19:22:47.714 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.714+0000 7f9304ff9640 10 client.4370.objecter ms_handle_connect 0x7f92f406ddf0 2024-03-25T19:22:47.719 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.719+0000 7f9304ff9640 1 -- 172.21.15.83:0/2508987900 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7f9300054b10 con 0x7f93081430c0 2024-03-25T19:22:47.719 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.719+0000 7f9304ff9640 10 client.4370.objecter ms_dispatch 0x7f9308000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:22:47.719 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.719+0000 7f9304ff9640 3 client.4370.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:22:47.719 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.719+0000 7f9304ff9640 3 client.4370.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:22:47.719 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.719+0000 7f9304ff9640 20 client.4370.objecter dump_active .. 0 homeless 2024-03-25T19:22:47.719 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.719+0000 7f9304ff9640 1 -- 172.21.15.83:0/2508987900 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9300050620 con 0x7f93081430c0 2024-03-25T19:22:47.719 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.719+0000 7f9304ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:47.719 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.719+0000 7f9304ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:22:47.968 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.968+0000 7f930e554640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:47.968 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.968+0000 7f930e554640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:47.968 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.968+0000 7f930e554640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:22:47.968 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:47.968+0000 7f930e554640 1 -- 172.21.15.83:0/2508987900 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f93080008d0 con 0x7f93081430c0 2024-03-25T19:22:48.000 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:48.000+0000 7f9304ff9640 1 -- 172.21.15.83:0/2508987900 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7f9300052550 con 0x7f93081430c0 2024-03-25T19:22:48.000 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:48.000+0000 7f9304ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:48.000 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:48.000+0000 7f9304ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:22:48.000 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:22:48.000 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:22:48.003 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:48.002+0000 7f930e554640 10 librados: watch_flush enter 2024-03-25T19:22:48.003 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:48.002+0000 7f930e554640 10 librados: watch_flush exit 2024-03-25T19:22:48.003 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:48.002+0000 7f930e554640 20 client.4370.objecter shutdown clearing up homeless session... 2024-03-25T19:22:48.003 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:48.002+0000 7f930e554640 10 client.4370.objecter successfully canceled tick 2024-03-25T19:22:48.003 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:48.003+0000 7f930e554640 1 -- 172.21.15.83:0/2508987900 >> v1:172.21.15.103:6800/345072327 conn(0x7f92f406ddf0 legacy=0x7f92f40702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:48.003 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:48.003+0000 7f930e554640 10 monclient: shutdown 2024-03-25T19:22:48.003 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:48.003+0000 7f930e554640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:48.003 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:48.003+0000 7f930e554640 1 -- 172.21.15.83:0/2508987900 >> v1:172.21.15.83:6790/0 conn(0x7f93081430c0 legacy=0x7f93081d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:48.003 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:48.003+0000 7f930e554640 1 -- 172.21.15.83:0/2508987900 shutdown_connections 2024-03-25T19:22:48.003 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:48.003+0000 7f930e554640 1 -- 172.21.15.83:0/2508987900 >> 172.21.15.83:0/2508987900 conn(0x7f93080b7ea0 msgr2=0x7f9308074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:48.003 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:48.003+0000 7f930e554640 1 -- 172.21.15.83:0/2508987900 shutdown_connections 2024-03-25T19:22:48.004 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:48.004+0000 7f930e554640 1 -- 172.21.15.83:0/2508987900 wait complete. 2024-03-25T19:22:48.004 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:48.004+0000 7f930e554640 1 librados: shutdown 2024-03-25T19:22:48.016 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:22:54.823 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:22:54.941 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.940+0000 7f0d1aaa8640 10 monclient: get_monmap_and_config 2024-03-25T19:22:54.941 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.940+0000 7f0d1aaa8640 10 monclient: build_initial_monmap 2024-03-25T19:22:54.941 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.940+0000 7f0d1aaa8640 10 monclient: monmap: 2024-03-25T19:22:54.941 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:22:54.941 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:22:54.942 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:22:54.942774+0000 2024-03-25T19:22:54.942 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:22:54.942774+0000 2024-03-25T19:22:54.942 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:54.942 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:22:54.942 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:22:54.942 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:22:54.942 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:22:54.942 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:54.942 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.941+0000 7f0d1aaa8640 1 Processor -- start 2024-03-25T19:22:54.942 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.941+0000 7f0d1aaa8640 1 -- start start 2024-03-25T19:22:54.943 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.941+0000 7f0d1aaa8640 10 monclient: init 2024-03-25T19:22:54.943 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.942+0000 7f0d1aaa8640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:54.943 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.942+0000 7f0d1aaa8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:22:54.943 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.942+0000 7f0d1aaa8640 10 monclient(hunting): picked mon.noname-c con 0x7f0d140b4d60 addr v1:172.21.15.103:6789/0 2024-03-25T19:22:54.943 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.942+0000 7f0d1aaa8640 10 monclient(hunting): picked mon.noname-a con 0x7f0d140b1730 addr v1:172.21.15.83:6789/0 2024-03-25T19:22:54.943 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.942+0000 7f0d1aaa8640 10 monclient(hunting): picked mon.noname-b con 0x7f0d1414be90 addr v1:172.21.15.83:6790/0 2024-03-25T19:22:54.943 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.942+0000 7f0d1aaa8640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d140af940 con 0x7f0d140b1730 2024-03-25T19:22:54.943 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.942+0000 7f0d1aaa8640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d140b0a70 con 0x7f0d140b4d60 2024-03-25T19:22:54.944 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.942+0000 7f0d1aaa8640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d140b0d50 con 0x7f0d1414be90 2024-03-25T19:22:54.944 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.942+0000 7f0d1aaa8640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:54.944 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.942+0000 7f0d1aaa8640 10 monclient(hunting): authenticate will time out at 6232.215530s 2024-03-25T19:22:54.953 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.952+0000 7f0d19045640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f0d1414be90 0x7f0d1414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:52330/0 (socket says 172.21.15.83:52330) 2024-03-25T19:22:54.953 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.952+0000 7f0d19045640 1 -- 172.21.15.83:0/2571625324 learned_addr learned my addr 172.21.15.83:0/2571625324 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:22:54.966 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.965+0000 7f0d137fe640 1 -- 172.21.15.83:0/2571625324 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3645983588 0 0) 0x7f0d140b0a70 con 0x7f0d140b4d60 2024-03-25T19:22:54.966 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.965+0000 7f0d137fe640 10 monclient(hunting): _negotiate 2024-03-25T19:22:54.966 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.965+0000 7f0d137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:54.966 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.965+0000 7f0d137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:54.966 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.965+0000 7f0d137fe640 10 monclient(hunting): my global_id is 4366 2024-03-25T19:22:54.967 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.965+0000 7f0d137fe640 1 -- 172.21.15.83:0/2571625324 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d04003520 con 0x7f0d140b4d60 2024-03-25T19:22:54.969 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.968+0000 7f0d137fe640 1 -- 172.21.15.83:0/2571625324 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1619020720 0 0) 0x7f0d140b0d50 con 0x7f0d1414be90 2024-03-25T19:22:54.969 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.968+0000 7f0d137fe640 10 monclient(hunting): _negotiate 2024-03-25T19:22:54.969 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.968+0000 7f0d137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:54.969 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.968+0000 7f0d137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:54.969 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.968+0000 7f0d137fe640 10 monclient(hunting): my global_id is 4373 2024-03-25T19:22:54.969 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.968+0000 7f0d137fe640 1 -- 172.21.15.83:0/2571625324 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d140b0a70 con 0x7f0d1414be90 2024-03-25T19:22:54.977 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.976+0000 7f0d137fe640 1 -- 172.21.15.83:0/2571625324 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 285659833 0 0) 0x7f0d140af940 con 0x7f0d140b1730 2024-03-25T19:22:54.977 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.976+0000 7f0d137fe640 10 monclient(hunting): _negotiate 2024-03-25T19:22:54.977 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.976+0000 7f0d137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:54.977 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.976+0000 7f0d137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:54.977 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.976+0000 7f0d137fe640 10 monclient(hunting): my global_id is 4362 2024-03-25T19:22:54.977 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.976+0000 7f0d137fe640 1 -- 172.21.15.83:0/2571625324 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d140b0d50 con 0x7f0d140b1730 2024-03-25T19:22:54.990 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.989+0000 7f0d137fe640 1 -- 172.21.15.83:0/2571625324 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1575828218 0 0) 0x7f0d140b0a70 con 0x7f0d1414be90 2024-03-25T19:22:54.990 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.989+0000 7f0d137fe640 1 -- 172.21.15.83:0/2571625324 >> v1:172.21.15.103:6789/0 conn(0x7f0d140b4d60 legacy=0x7f0d1413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:54.990 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.989+0000 7f0d137fe640 1 -- 172.21.15.83:0/2571625324 >> v1:172.21.15.83:6789/0 conn(0x7f0d140b1730 legacy=0x7f0d140b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:54.990 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.989+0000 7f0d137fe640 10 monclient: _finish_hunting 0 2024-03-25T19:22:54.990 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.989+0000 7f0d137fe640 1 monclient: found mon.noname-b 2024-03-25T19:22:54.990 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.989+0000 7f0d137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:54.990 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.989+0000 7f0d137fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.83:6790/0 2024-03-25T19:22:54.990 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.989+0000 7f0d137fe640 1 -- 172.21.15.83:0/2571625324 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d140b1030 con 0x7f0d1414be90 2024-03-25T19:22:54.990 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.989+0000 7f0d137fe640 10 monclient: _finish_auth 0 2024-03-25T19:22:54.990 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.989+0000 7f0d137fe640 10 monclient: _check_auth_tickets 2024-03-25T19:22:54.990 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.989+0000 7f0d137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:54.990 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.989+0000 7f0d137fe640 1 -- 172.21.15.83:0/2571625324 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f0d0c0019f0 con 0x7f0d1414be90 2024-03-25T19:22:54.991 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.989+0000 7f0d137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:54.991 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.989+0000 7f0d137fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:22:54.991 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.989+0000 7f0d137fe640 10 monclient: dump: 2024-03-25T19:22:54.991 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:22:54.991 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:22:54.991 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:54.991 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:54.991 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:54.991 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:22:54.991 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:22:54.991 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:22:54.991 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:22:54.991 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:54.991 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.989+0000 7f0d1aaa8640 5 monclient: authenticate success, global_id 4373 2024-03-25T19:22:54.991 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.989+0000 7f0d1aaa8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:54.999 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.999+0000 7f0d137fe640 1 -- 172.21.15.83:0/2571625324 <== mon.2 v1:172.21.15.83:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d0c001b50 con 0x7f0d1414be90 2024-03-25T19:22:55.000 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.999+0000 7f0d137fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:55.000 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.999+0000 7f0d137fe640 1 -- 172.21.15.83:0/2571625324 <== mon.2 v1:172.21.15.83:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f0d0c001f60 con 0x7f0d1414be90 2024-03-25T19:22:55.000 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.999+0000 7f0d137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:55.000 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.999+0000 7f0d137fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:22:55.000 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.999+0000 7f0d137fe640 10 monclient: dump: 2024-03-25T19:22:55.000 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:22:55.000 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:22:55.000 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:55.000 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:55.000 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:55.000 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:22:55.000 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:22:55.000 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:22:55.000 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:22:55.000 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:55.001 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.999+0000 7f0d1aaa8640 10 monclient: get_monmap_and_config success 2024-03-25T19:22:55.001 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.999+0000 7f0d1aaa8640 4 set_mon_vals no callback set 2024-03-25T19:22:55.002 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:54.999+0000 7f0d1aaa8640 10 monclient: shutdown 2024-03-25T19:22:55.002 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.000+0000 7f0d1aaa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:55.002 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.000+0000 7f0d1aaa8640 1 -- 172.21.15.83:0/2571625324 >> v1:172.21.15.83:6790/0 conn(0x7f0d1414be90 legacy=0x7f0d1414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:55.002 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.000+0000 7f0d1aaa8640 1 -- 172.21.15.83:0/2571625324 shutdown_connections 2024-03-25T19:22:55.003 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.000+0000 7f0d1aaa8640 1 -- 172.21.15.83:0/2571625324 >> 172.21.15.83:0/2571625324 conn(0x7f0d140617a0 msgr2=0x7f0d14061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:55.003 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.000+0000 7f0d1aaa8640 1 -- 172.21.15.83:0/2571625324 shutdown_connections 2024-03-25T19:22:55.003 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.000+0000 7f0d1aaa8640 1 -- 172.21.15.83:0/2571625324 wait complete. 2024-03-25T19:22:55.003 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.000+0000 7f0d1aaa8640 10 monclient: build_initial_monmap 2024-03-25T19:22:55.003 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.000+0000 7f0d1aaa8640 10 monclient: monmap: 2024-03-25T19:22:55.003 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:22:55.003 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:22:55.003 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:22:55.003 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:22:55.003 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:55.003 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:22:55.003 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:22:55.003 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:22:55.003 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:22:55.003 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:55.003 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.001+0000 7f0d1aaa8640 1 librados: starting msgr at 2024-03-25T19:22:55.004 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.001+0000 7f0d1aaa8640 1 librados: starting objecter 2024-03-25T19:22:55.004 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.001+0000 7f0d1aaa8640 1 Processor -- start 2024-03-25T19:22:55.004 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.001+0000 7f0d1aaa8640 1 -- start start 2024-03-25T19:22:55.004 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.001+0000 7f0d1aaa8640 1 librados: setting wanted keys 2024-03-25T19:22:55.004 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.001+0000 7f0d1aaa8640 1 librados: calling monclient init 2024-03-25T19:22:55.004 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.001+0000 7f0d1aaa8640 10 monclient: init 2024-03-25T19:22:55.004 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.001+0000 7f0d1aaa8640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:55.004 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.002+0000 7f0d1aaa8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:22:55.004 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.002+0000 7f0d1aaa8640 10 monclient(hunting): picked mon.noname-a con 0x7f0d140b1730 addr v1:172.21.15.83:6789/0 2024-03-25T19:22:55.004 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.002+0000 7f0d1aaa8640 10 monclient(hunting): picked mon.noname-b con 0x7f0d140b4d60 addr v1:172.21.15.103:6789/0 2024-03-25T19:22:55.004 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.002+0000 7f0d1aaa8640 10 monclient(hunting): picked mon.noname-c con 0x7f0d1414be90 addr v1:172.21.15.83:6790/0 2024-03-25T19:22:55.004 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.002+0000 7f0d1aaa8640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d141384b0 con 0x7f0d140b1730 2024-03-25T19:22:55.004 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.002+0000 7f0d1aaa8640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d141327e0 con 0x7f0d140b4d60 2024-03-25T19:22:55.004 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.002+0000 7f0d1aaa8640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d14137e70 con 0x7f0d1414be90 2024-03-25T19:22:55.005 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.002+0000 7f0d1aaa8640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:55.005 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.002+0000 7f0d1aaa8640 10 monclient(hunting): authenticate will time out at 6232.274860s 2024-03-25T19:22:55.006 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.006+0000 7f0d19045640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f0d1414be90 0x7f0d141320d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:52406/0 (socket says 172.21.15.83:52406) 2024-03-25T19:22:55.006 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.006+0000 7f0d19045640 1 -- 172.21.15.83:0/1038320009 learned_addr learned my addr 172.21.15.83:0/1038320009 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:22:55.013 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.013+0000 7f0d117fa640 10 client.?.objecter ms_handle_connect 0x7f0d140b1730 2024-03-25T19:22:55.013 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.013+0000 7f0d117fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:55.016 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.015+0000 7f0d117fa640 10 client.?.objecter ms_handle_connect 0x7f0d1414be90 2024-03-25T19:22:55.016 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.015+0000 7f0d117fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:55.017 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.016+0000 7f0d117fa640 10 client.?.objecter ms_handle_connect 0x7f0d140b4d60 2024-03-25T19:22:55.017 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.016+0000 7f0d117fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:55.020 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.020+0000 7f0d117fa640 1 -- 172.21.15.83:0/1038320009 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2804929964 0 0) 0x7f0d141327e0 con 0x7f0d140b4d60 2024-03-25T19:22:55.020 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.020+0000 7f0d117fa640 10 monclient(hunting): _negotiate 2024-03-25T19:22:55.020 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.020+0000 7f0d117fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:55.020 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.020+0000 7f0d117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:55.020 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.020+0000 7f0d117fa640 10 monclient(hunting): my global_id is 4369 2024-03-25T19:22:55.021 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.020+0000 7f0d117fa640 1 -- 172.21.15.83:0/1038320009 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cf8004eb0 con 0x7f0d140b4d60 2024-03-25T19:22:55.026 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.025+0000 7f0d117fa640 1 -- 172.21.15.83:0/1038320009 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 206416643 0 0) 0x7f0d14137e70 con 0x7f0d1414be90 2024-03-25T19:22:55.026 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.025+0000 7f0d117fa640 10 monclient(hunting): _negotiate 2024-03-25T19:22:55.026 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.025+0000 7f0d117fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:55.026 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.025+0000 7f0d117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:55.026 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.025+0000 7f0d117fa640 10 monclient(hunting): my global_id is 4376 2024-03-25T19:22:55.026 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.025+0000 7f0d117fa640 1 -- 172.21.15.83:0/1038320009 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d141327e0 con 0x7f0d1414be90 2024-03-25T19:22:55.043 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.043+0000 7f0d117fa640 1 -- 172.21.15.83:0/1038320009 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3749357961 0 0) 0x7f0d141384b0 con 0x7f0d140b1730 2024-03-25T19:22:55.043 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.043+0000 7f0d117fa640 10 monclient(hunting): _negotiate 2024-03-25T19:22:55.043 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.043+0000 7f0d117fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:55.043 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.043+0000 7f0d117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:55.043 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.043+0000 7f0d117fa640 10 monclient(hunting): my global_id is 4365 2024-03-25T19:22:55.044 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.043+0000 7f0d117fa640 1 -- 172.21.15.83:0/1038320009 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d14137e70 con 0x7f0d140b1730 2024-03-25T19:22:55.044 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.044+0000 7f0d117fa640 1 -- 172.21.15.83:0/1038320009 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 137523153 0 0) 0x7f0d141327e0 con 0x7f0d1414be90 2024-03-25T19:22:55.045 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.044+0000 7f0d117fa640 1 -- 172.21.15.83:0/1038320009 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d141384b0 con 0x7f0d1414be90 2024-03-25T19:22:55.045 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.044+0000 7f0d117fa640 1 -- 172.21.15.83:0/1038320009 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f0d0c001ea0 con 0x7f0d1414be90 2024-03-25T19:22:55.045 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.044+0000 7f0d117fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:55.045 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.044+0000 7f0d117fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:22:55.045 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.044+0000 7f0d117fa640 10 monclient(hunting): dump: 2024-03-25T19:22:55.045 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:22:55.045 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:22:55.045 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:55.045 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:55.045 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:55.045 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:22:55.045 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:22:55.045 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:22:55.045 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:22:55.046 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:55.046 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.044+0000 7f0d117fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:22:55.125 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.124+0000 7f0d117fa640 1 -- 172.21.15.83:0/1038320009 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1025831579 0 0) 0x7f0d14137e70 con 0x7f0d140b1730 2024-03-25T19:22:55.125 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.124+0000 7f0d117fa640 1 -- 172.21.15.83:0/1038320009 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d141327e0 con 0x7f0d140b1730 2024-03-25T19:22:55.126 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.124+0000 7f0d117fa640 1 -- 172.21.15.83:0/1038320009 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f0d080017b0 con 0x7f0d140b1730 2024-03-25T19:22:55.126 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.124+0000 7f0d117fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:55.126 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.124+0000 7f0d117fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:22:55.126 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.124+0000 7f0d117fa640 10 monclient(hunting): dump: 2024-03-25T19:22:55.126 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:22:55.126 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:22:55.126 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:55.126 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:55.126 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:55.126 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:22:55.126 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:22:55.126 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:22:55.126 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:22:55.126 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:55.143 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.142+0000 7f0d117fa640 1 -- 172.21.15.83:0/1038320009 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 471816363 0 0) 0x7f0d141327e0 con 0x7f0d140b1730 2024-03-25T19:22:55.143 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.142+0000 7f0d117fa640 1 -- 172.21.15.83:0/1038320009 >> v1:172.21.15.83:6790/0 conn(0x7f0d1414be90 legacy=0x7f0d141320d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:55.143 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.142+0000 7f0d117fa640 1 -- 172.21.15.83:0/1038320009 >> v1:172.21.15.103:6789/0 conn(0x7f0d140b4d60 legacy=0x7f0d141319c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:55.143 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.142+0000 7f0d117fa640 10 monclient: _finish_hunting 0 2024-03-25T19:22:55.144 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.142+0000 7f0d117fa640 1 monclient: found mon.a 2024-03-25T19:22:55.144 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.142+0000 7f0d117fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:55.144 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.142+0000 7f0d117fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:22:55.144 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.142+0000 7f0d117fa640 1 -- 172.21.15.83:0/1038320009 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d14138000 con 0x7f0d140b1730 2024-03-25T19:22:55.144 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.142+0000 7f0d117fa640 10 monclient: _finish_auth 0 2024-03-25T19:22:55.144 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.142+0000 7f0d117fa640 10 monclient: _check_auth_tickets 2024-03-25T19:22:55.144 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.142+0000 7f0d117fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:55.144 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.142+0000 7f0d1aaa8640 5 monclient: authenticate success, global_id 4365 2024-03-25T19:22:55.144 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.142+0000 7f0d1aaa8640 10 monclient: _renew_subs 2024-03-25T19:22:55.144 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.142+0000 7f0d1aaa8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:22:55.144 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.142+0000 7f0d1aaa8640 1 -- 172.21.15.83:0/1038320009 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d14145330 con 0x7f0d140b1730 2024-03-25T19:22:55.144 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.142+0000 7f0d1aaa8640 10 client.4365.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:22:55.144 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.142+0000 7f0d1aaa8640 10 monclient: _renew_subs 2024-03-25T19:22:55.144 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.142+0000 7f0d1aaa8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:22:55.144 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.142+0000 7f0d1aaa8640 1 -- 172.21.15.83:0/1038320009 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d14140380 con 0x7f0d140b1730 2024-03-25T19:22:55.144 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.142+0000 7f0d1aaa8640 1 librados: init done 2024-03-25T19:22:55.145 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.143+0000 7f0d1aaa8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:55.145 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.143+0000 7f0d1aaa8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:55.145 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.143+0000 7f0d1aaa8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:22:55.145 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.143+0000 7f0d1aaa8640 1 -- 172.21.15.83:0/1038320009 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d140b22c0 con 0x7f0d140b1730 2024-03-25T19:22:55.429 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.428+0000 7f0d117fa640 1 -- 172.21.15.83:0/1038320009 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d08001cc0 con 0x7f0d140b1730 2024-03-25T19:22:55.429 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.428+0000 7f0d117fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:55.429 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.428+0000 7f0d117fa640 1 -- 172.21.15.83:0/1038320009 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f0d080037b0 con 0x7f0d140b1730 2024-03-25T19:22:55.429 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.428+0000 7f0d117fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:55.429 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.428+0000 7f0d117fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:22:55.429 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.428+0000 7f0d117fa640 10 monclient: dump: 2024-03-25T19:22:55.429 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:22:55.429 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:22:55.429 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:55.429 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:22:55.429 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:55.429 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:22:55.430 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:22:55.430 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:22:55.430 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:22:55.430 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:22:55.430 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.428+0000 7f0d12ffd640 4 set_mon_vals no callback set 2024-03-25T19:22:55.430 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.429+0000 7f0d117fa640 1 -- 172.21.15.83:0/1038320009 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7f0d08019780 con 0x7f0d140b1730 2024-03-25T19:22:55.431 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.429+0000 7f0d13fff640 10 monclient: get_auth_request con 0x7f0cf806df80 auth_method 0 2024-03-25T19:22:55.431 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.430+0000 7f0d117fa640 10 client.4365.objecter ms_handle_connect 0x7f0cf806df80 2024-03-25T19:22:55.437 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.436+0000 7f0d117fa640 1 -- 172.21.15.83:0/1038320009 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7f0d08053530 con 0x7f0d140b1730 2024-03-25T19:22:55.437 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.436+0000 7f0d117fa640 10 client.4365.objecter ms_dispatch 0x7f0d14000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:22:55.437 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.436+0000 7f0d117fa640 3 client.4365.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:22:55.437 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.436+0000 7f0d117fa640 3 client.4365.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:22:55.437 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.436+0000 7f0d117fa640 20 client.4365.objecter dump_active .. 0 homeless 2024-03-25T19:22:55.437 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.436+0000 7f0d117fa640 1 -- 172.21.15.83:0/1038320009 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0d0804ee70 con 0x7f0d140b1730 2024-03-25T19:22:55.437 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.436+0000 7f0d117fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:55.437 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.436+0000 7f0d117fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:22:55.686 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.685+0000 7f0d1aaa8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:55.686 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.685+0000 7f0d1aaa8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:55.686 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.685+0000 7f0d1aaa8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:22:55.686 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.685+0000 7f0d1aaa8640 1 -- 172.21.15.83:0/1038320009 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d140627b0 con 0x7f0d140b1730 2024-03-25T19:22:55.722 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.721+0000 7f0d117fa640 1 -- 172.21.15.83:0/1038320009 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7f0d08050f70 con 0x7f0d140b1730 2024-03-25T19:22:55.722 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.721+0000 7f0d117fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:55.722 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.721+0000 7f0d117fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:22:55.722 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:22:55.723 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:22:55.724 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.723+0000 7f0d1aaa8640 10 librados: watch_flush enter 2024-03-25T19:22:55.724 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.723+0000 7f0d1aaa8640 10 librados: watch_flush exit 2024-03-25T19:22:55.725 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.723+0000 7f0d1aaa8640 20 client.4365.objecter shutdown clearing up homeless session... 2024-03-25T19:22:55.725 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.723+0000 7f0d1aaa8640 10 client.4365.objecter successfully canceled tick 2024-03-25T19:22:55.725 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.723+0000 7f0d1aaa8640 1 -- 172.21.15.83:0/1038320009 >> v1:172.21.15.103:6800/345072327 conn(0x7f0cf806df80 legacy=0x7f0cf8070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:55.725 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.724+0000 7f0d1aaa8640 10 monclient: shutdown 2024-03-25T19:22:55.725 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.724+0000 7f0d1aaa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:55.725 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.724+0000 7f0d1aaa8640 1 -- 172.21.15.83:0/1038320009 >> v1:172.21.15.83:6789/0 conn(0x7f0d140b1730 legacy=0x7f0d141dad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:55.725 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.724+0000 7f0d1aaa8640 1 -- 172.21.15.83:0/1038320009 shutdown_connections 2024-03-25T19:22:55.725 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.724+0000 7f0d1aaa8640 1 -- 172.21.15.83:0/1038320009 >> 172.21.15.83:0/1038320009 conn(0x7f0d140617a0 msgr2=0x7f0d1413c5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:55.725 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.724+0000 7f0d1aaa8640 1 -- 172.21.15.83:0/1038320009 shutdown_connections 2024-03-25T19:22:55.725 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.724+0000 7f0d1aaa8640 1 -- 172.21.15.83:0/1038320009 wait complete. 2024-03-25T19:22:55.725 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:22:55.724+0000 7f0d1aaa8640 1 librados: shutdown 2024-03-25T19:22:55.736 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:23:02.542 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:23:02.651 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.651+0000 7f1dc81af640 10 monclient: get_monmap_and_config 2024-03-25T19:23:02.651 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.651+0000 7f1dc81af640 10 monclient: build_initial_monmap 2024-03-25T19:23:02.651 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.651+0000 7f1dc81af640 10 monclient: monmap: 2024-03-25T19:23:02.651 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:23:02.651 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:23:02.651 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:23:02.652562+0000 2024-03-25T19:23:02.652 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:23:02.652562+0000 2024-03-25T19:23:02.652 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:02.652 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:23:02.652 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:23:02.652 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:23:02.652 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:23:02.652 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:02.652 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.652+0000 7f1dc81af640 1 Processor -- start 2024-03-25T19:23:02.653 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.652+0000 7f1dc81af640 1 -- start start 2024-03-25T19:23:02.653 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.652+0000 7f1dc81af640 10 monclient: init 2024-03-25T19:23:02.653 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.652+0000 7f1dc81af640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:02.653 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.652+0000 7f1dc81af640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:23:02.653 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.652+0000 7f1dc81af640 10 monclient(hunting): picked mon.noname-c con 0x7f1dc00a98f0 addr v1:172.21.15.103:6789/0 2024-03-25T19:23:02.653 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.652+0000 7f1dc81af640 10 monclient(hunting): picked mon.noname-a con 0x7f1dc013f710 addr v1:172.21.15.83:6789/0 2024-03-25T19:23:02.654 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.652+0000 7f1dc81af640 10 monclient(hunting): picked mon.noname-b con 0x7f1dc01430c0 addr v1:172.21.15.83:6790/0 2024-03-25T19:23:02.654 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.652+0000 7f1dc81af640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dc0142520 con 0x7f1dc013f710 2024-03-25T19:23:02.654 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.653+0000 7f1dc81af640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dc00b28a0 con 0x7f1dc00a98f0 2024-03-25T19:23:02.654 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.653+0000 7f1dc81af640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dc00b2a30 con 0x7f1dc01430c0 2024-03-25T19:23:02.654 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.653+0000 7f1dc81af640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:02.654 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.653+0000 7f1dc81af640 10 monclient(hunting): authenticate will time out at 6239.924771s 2024-03-25T19:23:02.654 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.653+0000 7f1dc674c640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f1dc01430c0 0x7f1dc014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:36244/0 (socket says 172.21.15.83:36244) 2024-03-25T19:23:02.654 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.653+0000 7f1dc674c640 1 -- 172.21.15.83:0/3003870304 learned_addr learned my addr 172.21.15.83:0/3003870304 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:23:02.685 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:03:44.612 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:02.691 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:03:44.618 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:02.714 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:03:44.641 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:02.739 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:03:44.649 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:02.740 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:03:44.667 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:02.743 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:03:44.653 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:02.744 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.744+0000 7f1dc4f49640 1 -- 172.21.15.83:0/3003870304 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3209320410 0 0) 0x7f1dc0142520 con 0x7f1dc013f710 2024-03-25T19:23:02.744 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.744+0000 7f1dc4f49640 10 monclient(hunting): _negotiate 2024-03-25T19:23:02.744 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.744+0000 7f1dc4f49640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:02.744 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.744+0000 7f1dc4f49640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:02.744 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.744+0000 7f1dc4f49640 10 monclient(hunting): my global_id is 4368 2024-03-25T19:23:02.744 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.744+0000 7f1dc4f49640 1 -- 172.21.15.83:0/3003870304 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dac003520 con 0x7f1dc013f710 2024-03-25T19:23:02.748 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.748+0000 7f1dc4f49640 1 -- 172.21.15.83:0/3003870304 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2275010928 0 0) 0x7f1dc00b28a0 con 0x7f1dc00a98f0 2024-03-25T19:23:02.748 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.748+0000 7f1dc4f49640 10 monclient(hunting): _negotiate 2024-03-25T19:23:02.748 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.748+0000 7f1dc4f49640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:02.748 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.748+0000 7f1dc4f49640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:02.748 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.748+0000 7f1dc4f49640 10 monclient(hunting): my global_id is 4372 2024-03-25T19:23:02.749 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.748+0000 7f1dc4f49640 1 -- 172.21.15.83:0/3003870304 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dc0142520 con 0x7f1dc00a98f0 2024-03-25T19:23:02.754 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.754+0000 7f1dc4f49640 1 -- 172.21.15.83:0/3003870304 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2728460436 0 0) 0x7f1dc00b2a30 con 0x7f1dc01430c0 2024-03-25T19:23:02.754 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.754+0000 7f1dc4f49640 10 monclient(hunting): _negotiate 2024-03-25T19:23:02.754 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.754+0000 7f1dc4f49640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:02.754 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.754+0000 7f1dc4f49640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:02.754 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.754+0000 7f1dc4f49640 10 monclient(hunting): my global_id is 4379 2024-03-25T19:23:02.754 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.754+0000 7f1dc4f49640 1 -- 172.21.15.83:0/3003870304 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dc00b28a0 con 0x7f1dc01430c0 2024-03-25T19:23:02.755 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:03:44.682 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:02.776 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.776+0000 7f1dc4f49640 1 -- 172.21.15.83:0/3003870304 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2507362889 0 0) 0x7f1dc00b28a0 con 0x7f1dc01430c0 2024-03-25T19:23:02.776 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.776+0000 7f1dc4f49640 1 -- 172.21.15.83:0/3003870304 >> v1:172.21.15.103:6789/0 conn(0x7f1dc00a98f0 legacy=0x7f1dc00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:02.776 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.776+0000 7f1dc4f49640 1 -- 172.21.15.83:0/3003870304 >> v1:172.21.15.83:6789/0 conn(0x7f1dc013f710 legacy=0x7f1dc01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:02.776 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.776+0000 7f1dc4f49640 10 monclient: _finish_hunting 0 2024-03-25T19:23:02.776 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.776+0000 7f1dc4f49640 1 monclient: found mon.noname-b 2024-03-25T19:23:02.776 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.776+0000 7f1dc4f49640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:02.776 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.776+0000 7f1dc4f49640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.83:6790/0 2024-03-25T19:23:02.776 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.776+0000 7f1dc4f49640 1 -- 172.21.15.83:0/3003870304 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dc00b44c0 con 0x7f1dc01430c0 2024-03-25T19:23:02.776 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.776+0000 7f1dc4f49640 10 monclient: _finish_auth 0 2024-03-25T19:23:02.776 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.776+0000 7f1dc4f49640 10 monclient: _check_auth_tickets 2024-03-25T19:23:02.776 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.776+0000 7f1dc4f49640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:02.776 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.776+0000 7f1dc4f49640 1 -- 172.21.15.83:0/3003870304 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f1db8001930 con 0x7f1dc01430c0 2024-03-25T19:23:02.776 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.776+0000 7f1dc81af640 5 monclient: authenticate success, global_id 4379 2024-03-25T19:23:02.777 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.776+0000 7f1dc81af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:23:02.777 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.776+0000 7f1dc4f49640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:02.777 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.776+0000 7f1dc4f49640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:23:02.777 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.776+0000 7f1dc4f49640 10 monclient: dump: 2024-03-25T19:23:02.777 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:23:02.777 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:23:02.777 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:02.777 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:02.777 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:02.777 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:23:02.777 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:23:02.777 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:23:02.777 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:23:02.777 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:02.777 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.776+0000 7f1dc81af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:23:02.793 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:03:44.703 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:02.801 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.802+0000 7f1dc4f49640 1 -- 172.21.15.83:0/3003870304 <== mon.2 v1:172.21.15.83:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1db8001a90 con 0x7f1dc01430c0 2024-03-25T19:23:02.802 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.802+0000 7f1dc4f49640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:02.802 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.802+0000 7f1dc4f49640 1 -- 172.21.15.83:0/3003870304 <== mon.2 v1:172.21.15.83:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f1db8001ee0 con 0x7f1dc01430c0 2024-03-25T19:23:02.802 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.802+0000 7f1dc4f49640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:02.802 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.802+0000 7f1dc4f49640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:23:02.802 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.802+0000 7f1dc4f49640 10 monclient: dump: 2024-03-25T19:23:02.802 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:23:02.802 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:23:02.802 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:02.802 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:02.802 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:02.802 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:23:02.802 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:23:02.802 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:23:02.802 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:23:02.802 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:02.803 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.802+0000 7f1dc81af640 10 monclient: get_monmap_and_config success 2024-03-25T19:23:02.803 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.802+0000 7f1dc81af640 4 set_mon_vals no callback set 2024-03-25T19:23:02.803 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:03:44.729 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:02.805 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.803+0000 7f1dc81af640 10 monclient: shutdown 2024-03-25T19:23:02.805 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.803+0000 7f1dc81af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:02.805 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.803+0000 7f1dc81af640 1 -- 172.21.15.83:0/3003870304 >> v1:172.21.15.83:6790/0 conn(0x7f1dc01430c0 legacy=0x7f1dc014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:02.805 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.803+0000 7f1dc81af640 1 -- 172.21.15.83:0/3003870304 shutdown_connections 2024-03-25T19:23:02.805 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.803+0000 7f1dc81af640 1 -- 172.21.15.83:0/3003870304 >> 172.21.15.83:0/3003870304 conn(0x7f1dc00b7ea0 msgr2=0x7f1dc00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:02.805 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.803+0000 7f1dc81af640 1 -- 172.21.15.83:0/3003870304 shutdown_connections 2024-03-25T19:23:02.805 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.803+0000 7f1dc81af640 1 -- 172.21.15.83:0/3003870304 wait complete. 2024-03-25T19:23:02.806 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.803+0000 7f1dc81af640 10 monclient: build_initial_monmap 2024-03-25T19:23:02.806 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.803+0000 7f1dc81af640 10 monclient: monmap: 2024-03-25T19:23:02.806 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:23:02.806 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:23:02.806 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:23:02.806 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:23:02.806 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:02.806 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:23:02.806 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:23:02.806 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:23:02.806 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:23:02.806 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:02.806 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.804+0000 7f1dc81af640 1 librados: starting msgr at 2024-03-25T19:23:02.806 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.804+0000 7f1dc81af640 1 librados: starting objecter 2024-03-25T19:23:02.807 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.804+0000 7f1dc81af640 1 Processor -- start 2024-03-25T19:23:02.807 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.804+0000 7f1dc81af640 1 -- start start 2024-03-25T19:23:02.807 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.804+0000 7f1dc81af640 1 librados: setting wanted keys 2024-03-25T19:23:02.807 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.804+0000 7f1dc81af640 1 librados: calling monclient init 2024-03-25T19:23:02.807 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.804+0000 7f1dc81af640 10 monclient: init 2024-03-25T19:23:02.807 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.805+0000 7f1dc81af640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:02.807 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.805+0000 7f1dc81af640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:23:02.807 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.805+0000 7f1dc81af640 10 monclient(hunting): picked mon.noname-b con 0x7f1dc00a98f0 addr v1:172.21.15.103:6789/0 2024-03-25T19:23:02.807 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.805+0000 7f1dc81af640 10 monclient(hunting): picked mon.noname-c con 0x7f1dc013f710 addr v1:172.21.15.83:6790/0 2024-03-25T19:23:02.807 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.805+0000 7f1dc81af640 10 monclient(hunting): picked mon.noname-a con 0x7f1dc01430c0 addr v1:172.21.15.83:6789/0 2024-03-25T19:23:02.807 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.805+0000 7f1dc81af640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dc0138b50 con 0x7f1dc01430c0 2024-03-25T19:23:02.807 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.805+0000 7f1dc81af640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dc0139950 con 0x7f1dc00a98f0 2024-03-25T19:23:02.807 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.805+0000 7f1dc81af640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dc0138070 con 0x7f1dc013f710 2024-03-25T19:23:02.807 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.805+0000 7f1dc81af640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:02.808 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.805+0000 7f1dc81af640 10 monclient(hunting): authenticate will time out at 6240.076984s 2024-03-25T19:23:02.816 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.816+0000 7f1dc5f4b640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f1dc00a98f0 0x7f1dc0137840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:33146/0 (socket says 172.21.15.83:33146) 2024-03-25T19:23:02.816 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.816+0000 7f1dc5f4b640 1 -- 172.21.15.83:0/2856065610 learned_addr learned my addr 172.21.15.83:0/2856065610 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:23:02.821 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.821+0000 7f1db67fc640 10 client.?.objecter ms_handle_connect 0x7f1dc00a98f0 2024-03-25T19:23:02.821 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.821+0000 7f1db67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:02.845 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.846+0000 7f1db67fc640 10 client.?.objecter ms_handle_connect 0x7f1dc01430c0 2024-03-25T19:23:02.845 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.846+0000 7f1db67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:02.847 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.848+0000 7f1db67fc640 10 client.?.objecter ms_handle_connect 0x7f1dc013f710 2024-03-25T19:23:02.847 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.848+0000 7f1db67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:02.849 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.850+0000 7f1db67fc640 1 -- 172.21.15.83:0/2856065610 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3981593004 0 0) 0x7f1dc0139950 con 0x7f1dc00a98f0 2024-03-25T19:23:02.849 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.850+0000 7f1db67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:23:02.849 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.850+0000 7f1db67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:02.849 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.850+0000 7f1db67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:02.850 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.850+0000 7f1db67fc640 10 monclient(hunting): my global_id is 4375 2024-03-25T19:23:02.850 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.850+0000 7f1db67fc640 1 -- 172.21.15.83:0/2856065610 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1da4004eb0 con 0x7f1dc00a98f0 2024-03-25T19:23:02.866 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.866+0000 7f1db67fc640 1 -- 172.21.15.83:0/2856065610 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1733980246 0 0) 0x7f1dc0138b50 con 0x7f1dc01430c0 2024-03-25T19:23:02.866 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.866+0000 7f1db67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:23:02.866 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.866+0000 7f1db67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:02.866 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.866+0000 7f1db67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:02.866 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.866+0000 7f1db67fc640 10 monclient(hunting): my global_id is 4371 2024-03-25T19:23:02.866 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.866+0000 7f1db67fc640 1 -- 172.21.15.83:0/2856065610 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dc0139950 con 0x7f1dc01430c0 2024-03-25T19:23:02.883 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.883+0000 7f1db67fc640 1 -- 172.21.15.83:0/2856065610 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 31192604 0 0) 0x7f1dc0139950 con 0x7f1dc01430c0 2024-03-25T19:23:02.883 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.883+0000 7f1db67fc640 1 -- 172.21.15.83:0/2856065610 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1dc0138b50 con 0x7f1dc01430c0 2024-03-25T19:23:02.883 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.883+0000 7f1db67fc640 1 -- 172.21.15.83:0/2856065610 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f1db8001da0 con 0x7f1dc01430c0 2024-03-25T19:23:02.883 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.883+0000 7f1db67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:02.884 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.883+0000 7f1db67fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:23:02.884 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.883+0000 7f1db67fc640 10 monclient(hunting): dump: 2024-03-25T19:23:02.884 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:23:02.884 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:23:02.884 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:02.884 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:02.884 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:02.884 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:23:02.884 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:23:02.884 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:23:02.884 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:23:02.884 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:02.884 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.883+0000 7f1db67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:23:02.886 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.886+0000 7f1db67fc640 1 -- 172.21.15.83:0/2856065610 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 717074396 0 0) 0x7f1da4004eb0 con 0x7f1dc00a98f0 2024-03-25T19:23:02.886 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.886+0000 7f1db67fc640 1 -- 172.21.15.83:0/2856065610 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1dc0139950 con 0x7f1dc00a98f0 2024-03-25T19:23:02.886 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.886+0000 7f1db67fc640 1 -- 172.21.15.83:0/2856065610 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f1dbc001a10 con 0x7f1dc00a98f0 2024-03-25T19:23:02.886 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.886+0000 7f1db67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:02.886 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.886+0000 7f1db67fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:23:02.886 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.886+0000 7f1db67fc640 10 monclient(hunting): dump: 2024-03-25T19:23:02.886 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:23:02.886 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:23:02.886 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:02.886 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:02.886 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:02.886 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:23:02.887 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:23:02.887 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:23:02.887 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:23:02.887 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:02.891 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.891+0000 7f1db67fc640 1 -- 172.21.15.83:0/2856065610 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 131599295 0 0) 0x7f1dc0138070 con 0x7f1dc013f710 2024-03-25T19:23:02.891 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.891+0000 7f1db67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:23:02.891 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.891+0000 7f1db67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:02.891 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.891+0000 7f1db67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:02.891 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.891+0000 7f1db67fc640 10 monclient(hunting): my global_id is 4382 2024-03-25T19:23:02.891 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.891+0000 7f1db67fc640 1 -- 172.21.15.83:0/2856065610 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1da4004eb0 con 0x7f1dc013f710 2024-03-25T19:23:02.896 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.896+0000 7f1db67fc640 1 -- 172.21.15.83:0/2856065610 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 988109931 0 0) 0x7f1dc0138b50 con 0x7f1dc01430c0 2024-03-25T19:23:02.896 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.896+0000 7f1db67fc640 1 -- 172.21.15.83:0/2856065610 >> v1:172.21.15.83:6790/0 conn(0x7f1dc013f710 legacy=0x7f1dc01da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:02.896 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.896+0000 7f1db67fc640 1 -- 172.21.15.83:0/2856065610 >> v1:172.21.15.103:6789/0 conn(0x7f1dc00a98f0 legacy=0x7f1dc0137840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:02.896 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.896+0000 7f1db67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:23:02.896 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.896+0000 7f1db67fc640 1 monclient: found mon.a 2024-03-25T19:23:02.896 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.896+0000 7f1db67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:02.896 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.896+0000 7f1db67fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:23:02.896 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.896+0000 7f1db67fc640 1 -- 172.21.15.83:0/2856065610 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dc0138200 con 0x7f1dc01430c0 2024-03-25T19:23:02.896 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.896+0000 7f1db67fc640 10 monclient: _finish_auth 0 2024-03-25T19:23:02.897 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.896+0000 7f1db67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:23:02.897 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.896+0000 7f1db67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:02.897 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.896+0000 7f1dc81af640 5 monclient: authenticate success, global_id 4371 2024-03-25T19:23:02.897 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.896+0000 7f1dc81af640 10 monclient: _renew_subs 2024-03-25T19:23:02.897 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.896+0000 7f1dc81af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:23:02.897 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.896+0000 7f1dc81af640 1 -- 172.21.15.83:0/2856065610 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dc01daf90 con 0x7f1dc01430c0 2024-03-25T19:23:02.897 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.896+0000 7f1dc81af640 10 client.4371.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:23:02.897 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.896+0000 7f1dc81af640 10 monclient: _renew_subs 2024-03-25T19:23:02.897 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.896+0000 7f1dc81af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:23:02.897 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.896+0000 7f1dc81af640 1 -- 172.21.15.83:0/2856065610 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dc01e34e0 con 0x7f1dc01430c0 2024-03-25T19:23:02.897 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.896+0000 7f1dc81af640 1 librados: init done 2024-03-25T19:23:02.898 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.897+0000 7f1dc81af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:02.898 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.897+0000 7f1dc81af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:02.898 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.897+0000 7f1dc81af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:23:02.898 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:02.897+0000 7f1dc81af640 1 -- 172.21.15.83:0/2856065610 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dc00627b0 con 0x7f1dc01430c0 2024-03-25T19:23:03.205 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:03.205+0000 7f1db67fc640 1 -- 172.21.15.83:0/2856065610 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1db8004b60 con 0x7f1dc01430c0 2024-03-25T19:23:03.205 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:03.205+0000 7f1db67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:03.205 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:03.205+0000 7f1db67fc640 1 -- 172.21.15.83:0/2856065610 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f1db8004e70 con 0x7f1dc01430c0 2024-03-25T19:23:03.205 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:03.205+0000 7f1db67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:03.205 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:03.205+0000 7f1db67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:23:03.205 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:03.205+0000 7f1db67fc640 10 monclient: dump: 2024-03-25T19:23:03.205 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:23:03.205 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:23:03.205 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:03.205 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:03.205 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:03.205 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:23:03.205 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:23:03.206 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:23:03.206 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:23:03.206 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:03.206 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:03.205+0000 7f1db7fff640 4 set_mon_vals no callback set 2024-03-25T19:23:03.206 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:03.206+0000 7f1db67fc640 1 -- 172.21.15.83:0/2856065610 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7f1db801ae40 con 0x7f1dc01430c0 2024-03-25T19:23:03.206 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:03.207+0000 7f1dc5f4b640 10 monclient: get_auth_request con 0x7f1da406df50 auth_method 0 2024-03-25T19:23:03.206 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:03.207+0000 7f1db67fc640 10 client.4371.objecter ms_handle_connect 0x7f1da406df50 2024-03-25T19:23:03.213 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:03.213+0000 7f1db67fc640 1 -- 172.21.15.83:0/2856065610 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7f1db8054a10 con 0x7f1dc01430c0 2024-03-25T19:23:03.213 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:03.213+0000 7f1db67fc640 10 client.4371.objecter ms_dispatch 0x7f1dc0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:23:03.213 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:03.213+0000 7f1db67fc640 3 client.4371.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:23:03.213 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:03.213+0000 7f1db67fc640 3 client.4371.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:23:03.213 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:03.213+0000 7f1db67fc640 20 client.4371.objecter dump_active .. 0 homeless 2024-03-25T19:23:03.213 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:03.213+0000 7f1db67fc640 1 -- 172.21.15.83:0/2856065610 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1db8050520 con 0x7f1dc01430c0 2024-03-25T19:23:03.213 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:03.213+0000 7f1db67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:03.213 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:03.213+0000 7f1db67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:23:03.466 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:03.467+0000 7f1dc81af640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:03.466 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:03.467+0000 7f1dc81af640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:03.467 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:03.467+0000 7f1dc81af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:23:03.467 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:03.467+0000 7f1dc81af640 1 -- 172.21.15.83:0/2856065610 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1dc0074cc0 con 0x7f1dc01430c0 2024-03-25T19:23:03.492 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:03.492+0000 7f1db67fc640 1 -- 172.21.15.83:0/2856065610 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7f1db8052450 con 0x7f1dc01430c0 2024-03-25T19:23:03.492 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:03.492+0000 7f1db67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:03.492 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:03.492+0000 7f1db67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:23:03.492 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:23:03.492 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:23:03.495 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:03.494+0000 7f1dc81af640 10 librados: watch_flush enter 2024-03-25T19:23:03.495 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:03.494+0000 7f1dc81af640 10 librados: watch_flush exit 2024-03-25T19:23:03.495 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:03.495+0000 7f1dc81af640 20 client.4371.objecter shutdown clearing up homeless session... 2024-03-25T19:23:03.495 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:03.495+0000 7f1dc81af640 10 client.4371.objecter successfully canceled tick 2024-03-25T19:23:03.495 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:03.495+0000 7f1dc81af640 1 -- 172.21.15.83:0/2856065610 >> v1:172.21.15.103:6800/345072327 conn(0x7f1da406df50 legacy=0x7f1da4070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:03.495 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:03.495+0000 7f1dc81af640 10 monclient: shutdown 2024-03-25T19:23:03.495 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:03.495+0000 7f1dc81af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:03.495 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:03.495+0000 7f1dc81af640 1 -- 172.21.15.83:0/2856065610 >> v1:172.21.15.83:6789/0 conn(0x7f1dc01430c0 legacy=0x7f1dc01e1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:03.495 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:03.495+0000 7f1dc81af640 1 -- 172.21.15.83:0/2856065610 shutdown_connections 2024-03-25T19:23:03.495 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:03.495+0000 7f1dc81af640 1 -- 172.21.15.83:0/2856065610 >> 172.21.15.83:0/2856065610 conn(0x7f1dc00b7ea0 msgr2=0x7f1dc0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:03.496 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:03.495+0000 7f1dc81af640 1 -- 172.21.15.83:0/2856065610 shutdown_connections 2024-03-25T19:23:03.496 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:03.495+0000 7f1dc81af640 1 -- 172.21.15.83:0/2856065610 wait complete. 2024-03-25T19:23:03.496 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:03.495+0000 7f1dc81af640 1 librados: shutdown 2024-03-25T19:23:03.504 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:23:10.311 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:23:10.425 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.425+0000 7f8cc1fb6640 10 monclient: get_monmap_and_config 2024-03-25T19:23:10.425 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.425+0000 7f8cc1fb6640 10 monclient: build_initial_monmap 2024-03-25T19:23:10.425 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.425+0000 7f8cc1fb6640 10 monclient: monmap: 2024-03-25T19:23:10.425 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:23:10.425 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:23:10.425 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:23:10.426670+0000 2024-03-25T19:23:10.426 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:23:10.426670+0000 2024-03-25T19:23:10.426 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:10.426 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:23:10.426 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:23:10.426 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:23:10.426 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:23:10.426 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:10.427 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.426+0000 7f8cc1fb6640 1 Processor -- start 2024-03-25T19:23:10.427 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.426+0000 7f8cc1fb6640 1 -- start start 2024-03-25T19:23:10.427 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.426+0000 7f8cc1fb6640 10 monclient: init 2024-03-25T19:23:10.427 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.427+0000 7f8cc1fb6640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:10.427 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.427+0000 7f8cc1fb6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:23:10.427 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.427+0000 7f8cc1fb6640 10 monclient(hunting): picked mon.noname-a con 0x7f8cbc0a98f0 addr v1:172.21.15.83:6789/0 2024-03-25T19:23:10.427 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.427+0000 7f8cc1fb6640 10 monclient(hunting): picked mon.noname-c con 0x7f8cbc13f710 addr v1:172.21.15.103:6789/0 2024-03-25T19:23:10.427 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.427+0000 7f8cc1fb6640 10 monclient(hunting): picked mon.noname-b con 0x7f8cbc1430c0 addr v1:172.21.15.83:6790/0 2024-03-25T19:23:10.428 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.427+0000 7f8cc1fb6640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cbc142520 con 0x7f8cbc0a98f0 2024-03-25T19:23:10.428 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.427+0000 7f8cc1fb6640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cbc0b28a0 con 0x7f8cbc13f710 2024-03-25T19:23:10.428 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.427+0000 7f8cc1fb6640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cbc0b2a30 con 0x7f8cbc1430c0 2024-03-25T19:23:10.428 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.427+0000 7f8cc1fb6640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:10.428 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.427+0000 7f8cc1fb6640 10 monclient(hunting): authenticate will time out at 6247.699193s 2024-03-25T19:23:10.432 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.432+0000 7f8cbbfff640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f8cbc1430c0 0x7f8cbc14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:49528/0 (socket says 172.21.15.83:49528) 2024-03-25T19:23:10.432 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.432+0000 7f8cbbfff640 1 -- 172.21.15.83:0/2587268604 learned_addr learned my addr 172.21.15.83:0/2587268604 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:23:10.442 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.442+0000 7f8cba7fc640 1 -- 172.21.15.83:0/2587268604 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1502788445 0 0) 0x7f8cbc0b2a30 con 0x7f8cbc1430c0 2024-03-25T19:23:10.442 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.442+0000 7f8cba7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:23:10.442 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.442+0000 7f8cba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:10.442 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.442+0000 7f8cba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:10.442 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.442+0000 7f8cba7fc640 10 monclient(hunting): my global_id is 4385 2024-03-25T19:23:10.442 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.442+0000 7f8cba7fc640 1 -- 172.21.15.83:0/2587268604 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ca4003520 con 0x7f8cbc1430c0 2024-03-25T19:23:10.452 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.452+0000 7f8cba7fc640 1 -- 172.21.15.83:0/2587268604 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3794600731 0 0) 0x7f8ca4003520 con 0x7f8cbc1430c0 2024-03-25T19:23:10.452 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.452+0000 7f8cba7fc640 1 -- 172.21.15.83:0/2587268604 >> v1:172.21.15.103:6789/0 conn(0x7f8cbc13f710 legacy=0x7f8cbc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:10.452 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.452+0000 7f8cba7fc640 1 -- 172.21.15.83:0/2587268604 >> v1:172.21.15.83:6789/0 conn(0x7f8cbc0a98f0 legacy=0x7f8cbc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:23:10.452 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.452+0000 7f8cba7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:23:10.452 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.452+0000 7f8cba7fc640 1 monclient: found mon.noname-b 2024-03-25T19:23:10.452 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.452+0000 7f8cba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:10.453 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.452+0000 7f8cba7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.83:6790/0 2024-03-25T19:23:10.453 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.452+0000 7f8cba7fc640 1 -- 172.21.15.83:0/2587268604 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cbc0b44c0 con 0x7f8cbc1430c0 2024-03-25T19:23:10.453 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.452+0000 7f8cba7fc640 10 monclient: _finish_auth 0 2024-03-25T19:23:10.453 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.452+0000 7f8cba7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:23:10.453 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.452+0000 7f8cba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:10.453 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.452+0000 7f8cba7fc640 1 -- 172.21.15.83:0/2587268604 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f8cb4001a10 con 0x7f8cbc1430c0 2024-03-25T19:23:10.453 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.452+0000 7f8cba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:10.453 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.452+0000 7f8cba7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:23:10.453 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.452+0000 7f8cba7fc640 10 monclient: dump: 2024-03-25T19:23:10.453 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:23:10.453 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:23:10.453 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:10.453 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:10.453 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:10.453 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:23:10.453 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:23:10.453 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:23:10.454 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:23:10.454 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:10.454 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.452+0000 7f8cc1fb6640 5 monclient: authenticate success, global_id 4385 2024-03-25T19:23:10.454 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.452+0000 7f8cc1fb6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:23:10.458 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.458+0000 7f8cba7fc640 1 -- 172.21.15.83:0/2587268604 <== mon.2 v1:172.21.15.83:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8cb4001be0 con 0x7f8cbc1430c0 2024-03-25T19:23:10.458 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.458+0000 7f8cba7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:10.458 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.458+0000 7f8cba7fc640 1 -- 172.21.15.83:0/2587268604 <== mon.2 v1:172.21.15.83:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f8cb4002030 con 0x7f8cbc1430c0 2024-03-25T19:23:10.458 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.458+0000 7f8cba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:10.458 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.458+0000 7f8cba7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:23:10.458 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.458+0000 7f8cba7fc640 10 monclient: dump: 2024-03-25T19:23:10.458 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:23:10.458 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:23:10.459 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:10.459 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:10.459 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:10.459 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:23:10.459 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:23:10.459 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:23:10.459 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:23:10.459 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:10.459 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.458+0000 7f8cc1fb6640 10 monclient: get_monmap_and_config success 2024-03-25T19:23:10.459 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.458+0000 7f8cc1fb6640 4 set_mon_vals no callback set 2024-03-25T19:23:10.461 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.459+0000 7f8cc1fb6640 10 monclient: shutdown 2024-03-25T19:23:10.461 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.459+0000 7f8cc1fb6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:10.461 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.459+0000 7f8cc1fb6640 1 -- 172.21.15.83:0/2587268604 >> v1:172.21.15.83:6790/0 conn(0x7f8cbc1430c0 legacy=0x7f8cbc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:10.461 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.459+0000 7f8cc1fb6640 1 -- 172.21.15.83:0/2587268604 shutdown_connections 2024-03-25T19:23:10.461 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.459+0000 7f8cc1fb6640 1 -- 172.21.15.83:0/2587268604 >> 172.21.15.83:0/2587268604 conn(0x7f8cbc0b7ea0 msgr2=0x7f8cbc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:10.461 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.459+0000 7f8cc1fb6640 1 -- 172.21.15.83:0/2587268604 shutdown_connections 2024-03-25T19:23:10.461 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.459+0000 7f8cc1fb6640 1 -- 172.21.15.83:0/2587268604 wait complete. 2024-03-25T19:23:10.461 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.460+0000 7f8cc1fb6640 10 monclient: build_initial_monmap 2024-03-25T19:23:10.461 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.460+0000 7f8cc1fb6640 10 monclient: monmap: 2024-03-25T19:23:10.461 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:23:10.461 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:23:10.461 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:23:10.461 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:23:10.461 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:10.462 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:23:10.462 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:23:10.462 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:23:10.462 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:23:10.462 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:10.462 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.461+0000 7f8cc1fb6640 1 librados: starting msgr at 2024-03-25T19:23:10.462 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.461+0000 7f8cc1fb6640 1 librados: starting objecter 2024-03-25T19:23:10.462 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.461+0000 7f8cc1fb6640 1 Processor -- start 2024-03-25T19:23:10.462 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.461+0000 7f8cc1fb6640 1 -- start start 2024-03-25T19:23:10.462 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.461+0000 7f8cc1fb6640 1 librados: setting wanted keys 2024-03-25T19:23:10.462 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.461+0000 7f8cc1fb6640 1 librados: calling monclient init 2024-03-25T19:23:10.462 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.461+0000 7f8cc1fb6640 10 monclient: init 2024-03-25T19:23:10.464 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.462+0000 7f8cc1fb6640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:10.464 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.462+0000 7f8cc1fb6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:23:10.464 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.462+0000 7f8cc1fb6640 10 monclient(hunting): picked mon.noname-a con 0x7f8cbc0a98f0 addr v1:172.21.15.83:6789/0 2024-03-25T19:23:10.464 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.462+0000 7f8cc1fb6640 10 monclient(hunting): picked mon.noname-c con 0x7f8cbc13f710 addr v1:172.21.15.83:6790/0 2024-03-25T19:23:10.464 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.462+0000 7f8cc1fb6640 10 monclient(hunting): picked mon.noname-b con 0x7f8cbc1430c0 addr v1:172.21.15.103:6789/0 2024-03-25T19:23:10.464 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.462+0000 7f8cc1fb6640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cbc0abcf0 con 0x7f8cbc0a98f0 2024-03-25T19:23:10.464 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.462+0000 7f8cc1fb6640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cbc1d51f0 con 0x7f8cbc1430c0 2024-03-25T19:23:10.464 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.462+0000 7f8cc1fb6640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cbc1d83d0 con 0x7f8cbc13f710 2024-03-25T19:23:10.464 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.462+0000 7f8cc1fb6640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:10.464 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.462+0000 7f8cc1fb6640 10 monclient(hunting): authenticate will time out at 6247.734176s 2024-03-25T19:23:10.465 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.463+0000 7f8cbbfff640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f8cbc1430c0 0x7f8cbc1e1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:36576/0 (socket says 172.21.15.83:36576) 2024-03-25T19:23:10.465 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.463+0000 7f8cbbfff640 1 -- 172.21.15.83:0/3857812643 learned_addr learned my addr 172.21.15.83:0/3857812643 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:23:10.465 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.464+0000 7f8ca3fff640 10 client.?.objecter ms_handle_connect 0x7f8cbc1430c0 2024-03-25T19:23:10.465 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.464+0000 7f8ca3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:10.469 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.469+0000 7f8ca3fff640 10 client.?.objecter ms_handle_connect 0x7f8cbc0a98f0 2024-03-25T19:23:10.469 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.469+0000 7f8ca3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:10.474 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.474+0000 7f8ca3fff640 1 -- 172.21.15.83:0/3857812643 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 171599570 0 0) 0x7f8cbc1d51f0 con 0x7f8cbc1430c0 2024-03-25T19:23:10.474 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.474+0000 7f8ca3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:23:10.474 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.474+0000 7f8ca3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:10.474 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.474+0000 7f8ca3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:10.474 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.474+0000 7f8ca3fff640 10 monclient(hunting): my global_id is 4381 2024-03-25T19:23:10.474 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.474+0000 7f8ca3fff640 1 -- 172.21.15.83:0/3857812643 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ca8004eb0 con 0x7f8cbc1430c0 2024-03-25T19:23:10.480 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.481+0000 7f8ca3fff640 10 client.?.objecter ms_handle_connect 0x7f8cbc13f710 2024-03-25T19:23:10.480 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.481+0000 7f8ca3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:10.491 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.492+0000 7f8ca3fff640 1 -- 172.21.15.83:0/3857812643 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 301475113 0 0) 0x7f8cbc0abcf0 con 0x7f8cbc0a98f0 2024-03-25T19:23:10.492 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.492+0000 7f8ca3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:23:10.492 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.492+0000 7f8ca3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:10.492 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.492+0000 7f8ca3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:10.492 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.492+0000 7f8ca3fff640 10 monclient(hunting): my global_id is 4374 2024-03-25T19:23:10.492 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.492+0000 7f8ca3fff640 1 -- 172.21.15.83:0/3857812643 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cbc1d51f0 con 0x7f8cbc0a98f0 2024-03-25T19:23:10.499 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.500+0000 7f8ca3fff640 1 -- 172.21.15.83:0/3857812643 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1014160312 0 0) 0x7f8cbc1d83d0 con 0x7f8cbc13f710 2024-03-25T19:23:10.499 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.500+0000 7f8ca3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:23:10.499 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.500+0000 7f8ca3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:10.499 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.500+0000 7f8ca3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:10.500 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.500+0000 7f8ca3fff640 10 monclient(hunting): my global_id is 4388 2024-03-25T19:23:10.500 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.500+0000 7f8ca3fff640 1 -- 172.21.15.83:0/3857812643 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cbc0abcf0 con 0x7f8cbc13f710 2024-03-25T19:23:10.508 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.509+0000 7f8ca3fff640 1 -- 172.21.15.83:0/3857812643 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3277775583 0 0) 0x7f8ca8004eb0 con 0x7f8cbc1430c0 2024-03-25T19:23:10.508 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.509+0000 7f8ca3fff640 1 -- 172.21.15.83:0/3857812643 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8cbc1d83d0 con 0x7f8cbc1430c0 2024-03-25T19:23:10.508 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.509+0000 7f8ca3fff640 1 -- 172.21.15.83:0/3857812643 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f8cb4001ef0 con 0x7f8cbc1430c0 2024-03-25T19:23:10.509 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.509+0000 7f8ca3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:10.509 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.509+0000 7f8ca3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:23:10.509 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.509+0000 7f8ca3fff640 10 monclient(hunting): dump: 2024-03-25T19:23:10.509 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:23:10.509 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:23:10.509 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:10.509 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:10.509 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:10.509 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:23:10.509 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:23:10.509 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:23:10.509 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:23:10.509 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:10.509 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.509+0000 7f8ca3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:23:10.514 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.515+0000 7f8ca3fff640 1 -- 172.21.15.83:0/3857812643 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2615612941 0 0) 0x7f8cbc1d51f0 con 0x7f8cbc0a98f0 2024-03-25T19:23:10.515 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.515+0000 7f8ca3fff640 1 -- 172.21.15.83:0/3857812643 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ca8004eb0 con 0x7f8cbc0a98f0 2024-03-25T19:23:10.515 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.515+0000 7f8ca3fff640 1 -- 172.21.15.83:0/3857812643 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f8cac001510 con 0x7f8cbc0a98f0 2024-03-25T19:23:10.515 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.515+0000 7f8ca3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:10.515 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.515+0000 7f8ca3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:23:10.515 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.515+0000 7f8ca3fff640 10 monclient(hunting): dump: 2024-03-25T19:23:10.515 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:23:10.515 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:23:10.515 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:10.515 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:10.515 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:10.515 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:23:10.515 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:23:10.515 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:23:10.515 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:23:10.515 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:10.524 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.523+0000 7f8ca3fff640 1 -- 172.21.15.83:0/3857812643 <== mon.1 v1:172.21.15.103:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 168644765 0 0) 0x7f8cbc1d83d0 con 0x7f8cbc1430c0 2024-03-25T19:23:10.524 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.523+0000 7f8ca3fff640 1 -- 172.21.15.83:0/3857812643 >> v1:172.21.15.83:6790/0 conn(0x7f8cbc13f710 legacy=0x7f8cbc138730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:10.524 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.523+0000 7f8ca3fff640 1 -- 172.21.15.83:0/3857812643 >> v1:172.21.15.83:6789/0 conn(0x7f8cbc0a98f0 legacy=0x7f8cbc1d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:10.524 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.523+0000 7f8ca3fff640 10 monclient: _finish_hunting 0 2024-03-25T19:23:10.524 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.523+0000 7f8ca3fff640 1 monclient: found mon.b 2024-03-25T19:23:10.524 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.523+0000 7f8ca3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:10.524 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.523+0000 7f8ca3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:23:10.524 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.523+0000 7f8ca3fff640 1 -- 172.21.15.83:0/3857812643 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cbc138e40 con 0x7f8cbc1430c0 2024-03-25T19:23:10.524 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.523+0000 7f8ca3fff640 10 monclient: _finish_auth 0 2024-03-25T19:23:10.524 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.523+0000 7f8ca3fff640 10 monclient: _check_auth_tickets 2024-03-25T19:23:10.524 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.523+0000 7f8ca3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:10.524 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.523+0000 7f8cc1fb6640 5 monclient: authenticate success, global_id 4381 2024-03-25T19:23:10.525 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.524+0000 7f8cc1fb6640 10 monclient: _renew_subs 2024-03-25T19:23:10.525 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.524+0000 7f8cc1fb6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:23:10.525 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.524+0000 7f8cc1fb6640 1 -- 172.21.15.83:0/3857812643 --> v1:172.21.15.103:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cbc137f70 con 0x7f8cbc1430c0 2024-03-25T19:23:10.525 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.524+0000 7f8cc1fb6640 10 client.4381.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:23:10.525 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.524+0000 7f8cc1fb6640 10 monclient: _renew_subs 2024-03-25T19:23:10.525 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.524+0000 7f8cc1fb6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:23:10.525 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.524+0000 7f8cc1fb6640 1 -- 172.21.15.83:0/3857812643 --> v1:172.21.15.103:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cbc1e3550 con 0x7f8cbc1430c0 2024-03-25T19:23:10.525 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.524+0000 7f8cc1fb6640 1 librados: init done 2024-03-25T19:23:10.526 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.525+0000 7f8cc1fb6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:10.526 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.525+0000 7f8cc1fb6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:10.526 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.525+0000 7f8cc1fb6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:23:10.526 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.525+0000 7f8cc1fb6640 1 -- 172.21.15.83:0/3857812643 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cbc0627b0 con 0x7f8cbc1430c0 2024-03-25T19:23:10.824 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.824+0000 7f8ca3fff640 1 -- 172.21.15.83:0/3857812643 <== mon.1 v1:172.21.15.103:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8cb4004e10 con 0x7f8cbc1430c0 2024-03-25T19:23:10.824 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.824+0000 7f8ca3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:10.824 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.824+0000 7f8ca3fff640 1 -- 172.21.15.83:0/3857812643 <== mon.1 v1:172.21.15.103:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f8cb4005120 con 0x7f8cbc1430c0 2024-03-25T19:23:10.824 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.824+0000 7f8ca3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:10.824 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.824+0000 7f8ca3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:23:10.824 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.824+0000 7f8ca3fff640 10 monclient: dump: 2024-03-25T19:23:10.824 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:23:10.824 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:23:10.824 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:10.824 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:10.824 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:10.825 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:23:10.825 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:23:10.825 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:23:10.825 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:23:10.825 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:10.825 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.824+0000 7f8cb9ffb640 4 set_mon_vals no callback set 2024-03-25T19:23:10.825 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.824+0000 7f8ca3fff640 1 -- 172.21.15.83:0/3857812643 <== mon.1 v1:172.21.15.103:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7f8cb401b0f0 con 0x7f8cbc1430c0 2024-03-25T19:23:10.825 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.825+0000 7f8cbb7fe640 10 monclient: get_auth_request con 0x7f8ca806dec0 auth_method 0 2024-03-25T19:23:10.825 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.825+0000 7f8ca3fff640 10 client.4381.objecter ms_handle_connect 0x7f8ca806dec0 2024-03-25T19:23:10.831 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.831+0000 7f8ca3fff640 1 -- 172.21.15.83:0/3857812643 <== mon.1 v1:172.21.15.103:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7f8cb4054c50 con 0x7f8cbc1430c0 2024-03-25T19:23:10.831 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.831+0000 7f8ca3fff640 10 client.4381.objecter ms_dispatch 0x7f8cbc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:23:10.831 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.831+0000 7f8ca3fff640 3 client.4381.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:23:10.831 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.831+0000 7f8ca3fff640 3 client.4381.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:23:10.831 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.831+0000 7f8ca3fff640 20 client.4381.objecter dump_active .. 0 homeless 2024-03-25T19:23:10.831 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.832+0000 7f8ca3fff640 1 -- 172.21.15.83:0/3857812643 <== mon.1 v1:172.21.15.103:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8cb4050760 con 0x7f8cbc1430c0 2024-03-25T19:23:10.831 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.832+0000 7f8ca3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:10.832 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:10.832+0000 7f8ca3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:23:11.081 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:11.081+0000 7f8cc1fb6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:11.081 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:11.081+0000 7f8cc1fb6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:11.081 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:11.081+0000 7f8cc1fb6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:23:11.081 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:11.081+0000 7f8cc1fb6640 1 -- 172.21.15.83:0/3857812643 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8cbc0b71a0 con 0x7f8cbc1430c0 2024-03-25T19:23:11.107 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:11.107+0000 7f8ca3fff640 1 -- 172.21.15.83:0/3857812643 <== mon.1 v1:172.21.15.103:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7f8cb4052690 con 0x7f8cbc1430c0 2024-03-25T19:23:11.107 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:11.107+0000 7f8ca3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:11.107 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:11.107+0000 7f8ca3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:23:11.108 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:23:11.108 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:23:11.110 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:11.110+0000 7f8cc1fb6640 10 librados: watch_flush enter 2024-03-25T19:23:11.110 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:11.110+0000 7f8cc1fb6640 10 librados: watch_flush exit 2024-03-25T19:23:11.110 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:11.110+0000 7f8cc1fb6640 20 client.4381.objecter shutdown clearing up homeless session... 2024-03-25T19:23:11.110 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:11.110+0000 7f8cc1fb6640 10 client.4381.objecter successfully canceled tick 2024-03-25T19:23:11.110 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:11.110+0000 7f8cc1fb6640 1 -- 172.21.15.83:0/3857812643 >> v1:172.21.15.103:6800/345072327 conn(0x7f8ca806dec0 legacy=0x7f8ca8070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:11.110 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:11.110+0000 7f8cc1fb6640 10 monclient: shutdown 2024-03-25T19:23:11.110 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:11.110+0000 7f8cc1fb6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:11.110 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:11.110+0000 7f8cc1fb6640 1 -- 172.21.15.83:0/3857812643 >> v1:172.21.15.103:6789/0 conn(0x7f8cbc1430c0 legacy=0x7f8cbc1e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:11.111 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:11.110+0000 7f8cc1fb6640 1 -- 172.21.15.83:0/3857812643 shutdown_connections 2024-03-25T19:23:11.111 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:11.110+0000 7f8cc1fb6640 1 -- 172.21.15.83:0/3857812643 >> 172.21.15.83:0/3857812643 conn(0x7f8cbc0b7ea0 msgr2=0x7f8cbc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:11.111 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:11.110+0000 7f8cc1fb6640 1 -- 172.21.15.83:0/3857812643 shutdown_connections 2024-03-25T19:23:11.111 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:11.110+0000 7f8cc1fb6640 1 -- 172.21.15.83:0/3857812643 wait complete. 2024-03-25T19:23:11.111 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:11.110+0000 7f8cc1fb6640 1 librados: shutdown 2024-03-25T19:23:11.121 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:23:17.928 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:23:18.041 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.042+0000 7f64aec7b640 10 monclient: get_monmap_and_config 2024-03-25T19:23:18.042 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.042+0000 7f64aec7b640 10 monclient: build_initial_monmap 2024-03-25T19:23:18.042 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.042+0000 7f64aec7b640 10 monclient: monmap: 2024-03-25T19:23:18.042 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:23:18.042 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:23:18.042 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:23:18.043293+0000 2024-03-25T19:23:18.042 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:23:18.043293+0000 2024-03-25T19:23:18.042 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:18.042 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:23:18.042 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:23:18.042 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:23:18.042 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:23:18.042 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:18.044 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.043+0000 7f64aec7b640 1 Processor -- start 2024-03-25T19:23:18.044 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.043+0000 7f64aec7b640 1 -- start start 2024-03-25T19:23:18.044 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.043+0000 7f64aec7b640 10 monclient: init 2024-03-25T19:23:18.044 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.043+0000 7f64aec7b640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:18.044 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.043+0000 7f64aec7b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:23:18.044 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.043+0000 7f64aec7b640 10 monclient(hunting): picked mon.noname-c con 0x7f64a80a98f0 addr v1:172.21.15.103:6789/0 2024-03-25T19:23:18.044 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.043+0000 7f64aec7b640 10 monclient(hunting): picked mon.noname-b con 0x7f64a813f710 addr v1:172.21.15.83:6790/0 2024-03-25T19:23:18.044 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.043+0000 7f64aec7b640 10 monclient(hunting): picked mon.noname-a con 0x7f64a81430c0 addr v1:172.21.15.83:6789/0 2024-03-25T19:23:18.044 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.043+0000 7f64aec7b640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64a8142520 con 0x7f64a81430c0 2024-03-25T19:23:18.044 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.043+0000 7f64aec7b640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64a80b28a0 con 0x7f64a80a98f0 2024-03-25T19:23:18.044 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.043+0000 7f64aec7b640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64a80b2a30 con 0x7f64a813f710 2024-03-25T19:23:18.044 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.043+0000 7f64aec7b640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:18.044 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.043+0000 7f64aec7b640 10 monclient(hunting): authenticate will time out at 6255.315652s 2024-03-25T19:23:18.052 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.052+0000 7f64aca17640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f64a80a98f0 0x7f64a80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:45254/0 (socket says 172.21.15.83:45254) 2024-03-25T19:23:18.052 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.052+0000 7f64aca17640 1 -- 172.21.15.83:0/2188411653 learned_addr learned my addr 172.21.15.83:0/2188411653 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:23:18.067 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.067+0000 7f64a77fe640 1 -- 172.21.15.83:0/2188411653 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4051629005 0 0) 0x7f64a80b2a30 con 0x7f64a813f710 2024-03-25T19:23:18.067 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.067+0000 7f64a77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:23:18.067 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.067+0000 7f64a77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:18.067 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.067+0000 7f64a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:18.067 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.067+0000 7f64a77fe640 10 monclient(hunting): my global_id is 4391 2024-03-25T19:23:18.067 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.067+0000 7f64a77fe640 1 -- 172.21.15.83:0/2188411653 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6498003520 con 0x7f64a813f710 2024-03-25T19:23:18.072 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.072+0000 7f64a77fe640 1 -- 172.21.15.83:0/2188411653 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2335912403 0 0) 0x7f64a80b28a0 con 0x7f64a80a98f0 2024-03-25T19:23:18.072 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.072+0000 7f64a77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:23:18.072 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.072+0000 7f64a77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:18.072 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.072+0000 7f64a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:18.072 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.072+0000 7f64a77fe640 10 monclient(hunting): my global_id is 4384 2024-03-25T19:23:18.072 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.072+0000 7f64a77fe640 1 -- 172.21.15.83:0/2188411653 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64a80b2a30 con 0x7f64a80a98f0 2024-03-25T19:23:18.082 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.082+0000 7f64a77fe640 1 -- 172.21.15.83:0/2188411653 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 607396991 0 0) 0x7f64a8142520 con 0x7f64a81430c0 2024-03-25T19:23:18.082 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.082+0000 7f64a77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:23:18.082 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.082+0000 7f64a77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:18.082 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.082+0000 7f64a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:18.082 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.082+0000 7f64a77fe640 10 monclient(hunting): my global_id is 4377 2024-03-25T19:23:18.083 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.082+0000 7f64a77fe640 1 -- 172.21.15.83:0/2188411653 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64a80b28a0 con 0x7f64a81430c0 2024-03-25T19:23:18.083 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.082+0000 7f64a77fe640 1 -- 172.21.15.83:0/2188411653 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1106296071 0 0) 0x7f6498003520 con 0x7f64a813f710 2024-03-25T19:23:18.083 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.082+0000 7f64a77fe640 1 -- 172.21.15.83:0/2188411653 >> v1:172.21.15.103:6789/0 conn(0x7f64a80a98f0 legacy=0x7f64a80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:18.083 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.082+0000 7f64a77fe640 1 -- 172.21.15.83:0/2188411653 >> v1:172.21.15.83:6789/0 conn(0x7f64a81430c0 legacy=0x7f64a814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:18.083 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.082+0000 7f64a77fe640 10 monclient: _finish_hunting 0 2024-03-25T19:23:18.083 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.082+0000 7f64a77fe640 1 monclient: found mon.noname-b 2024-03-25T19:23:18.083 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.082+0000 7f64a77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:18.083 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.082+0000 7f64a77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.83:6790/0 2024-03-25T19:23:18.083 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.082+0000 7f64a77fe640 1 -- 172.21.15.83:0/2188411653 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a80b44c0 con 0x7f64a813f710 2024-03-25T19:23:18.083 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.082+0000 7f64a77fe640 10 monclient: _finish_auth 0 2024-03-25T19:23:18.083 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.082+0000 7f64a77fe640 10 monclient: _check_auth_tickets 2024-03-25T19:23:18.083 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.082+0000 7f64a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:18.083 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.082+0000 7f64a77fe640 1 -- 172.21.15.83:0/2188411653 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f6494001690 con 0x7f64a813f710 2024-03-25T19:23:18.083 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.082+0000 7f64aec7b640 5 monclient: authenticate success, global_id 4391 2024-03-25T19:23:18.083 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.082+0000 7f64aec7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:23:18.084 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.082+0000 7f64a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:18.084 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.082+0000 7f64a77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:23:18.084 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.082+0000 7f64a77fe640 10 monclient: dump: 2024-03-25T19:23:18.084 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:23:18.084 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:23:18.084 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:18.084 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:18.084 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:18.084 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:23:18.084 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:23:18.084 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:23:18.084 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:23:18.084 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:18.084 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.082+0000 7f64aec7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:23:18.092 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.092+0000 7f64a77fe640 1 -- 172.21.15.83:0/2188411653 <== mon.2 v1:172.21.15.83:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64940017f0 con 0x7f64a813f710 2024-03-25T19:23:18.092 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.092+0000 7f64a77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:18.092 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.092+0000 7f64a77fe640 1 -- 172.21.15.83:0/2188411653 <== mon.2 v1:172.21.15.83:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f6494001c00 con 0x7f64a813f710 2024-03-25T19:23:18.092 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.092+0000 7f64aec7b640 10 monclient: get_monmap_and_config success 2024-03-25T19:23:18.092 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.092+0000 7f64aec7b640 4 set_mon_vals no callback set 2024-03-25T19:23:18.092 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.092+0000 7f64a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:18.092 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.092+0000 7f64a77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:23:18.092 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.092+0000 7f64a77fe640 10 monclient: dump: 2024-03-25T19:23:18.092 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:23:18.092 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:23:18.092 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:18.092 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:18.093 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:18.093 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:23:18.093 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:23:18.093 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:23:18.093 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:23:18.093 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:18.094 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.093+0000 7f64aec7b640 10 monclient: shutdown 2024-03-25T19:23:18.094 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.093+0000 7f64aec7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:18.094 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.093+0000 7f64aec7b640 1 -- 172.21.15.83:0/2188411653 >> v1:172.21.15.83:6790/0 conn(0x7f64a813f710 legacy=0x7f64a81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:18.094 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.093+0000 7f64aec7b640 1 -- 172.21.15.83:0/2188411653 shutdown_connections 2024-03-25T19:23:18.094 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.093+0000 7f64aec7b640 1 -- 172.21.15.83:0/2188411653 >> 172.21.15.83:0/2188411653 conn(0x7f64a80b7ea0 msgr2=0x7f64a80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:18.095 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.093+0000 7f64aec7b640 1 -- 172.21.15.83:0/2188411653 shutdown_connections 2024-03-25T19:23:18.095 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.093+0000 7f64aec7b640 1 -- 172.21.15.83:0/2188411653 wait complete. 2024-03-25T19:23:18.095 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.094+0000 7f64aec7b640 10 monclient: build_initial_monmap 2024-03-25T19:23:18.095 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.094+0000 7f64aec7b640 10 monclient: monmap: 2024-03-25T19:23:18.095 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:23:18.095 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:23:18.095 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:23:18.095 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:23:18.095 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:18.095 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:23:18.095 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:23:18.095 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:23:18.095 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:23:18.095 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:18.096 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.094+0000 7f64aec7b640 1 librados: starting msgr at 2024-03-25T19:23:18.096 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.094+0000 7f64aec7b640 1 librados: starting objecter 2024-03-25T19:23:18.096 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.094+0000 7f64aec7b640 1 Processor -- start 2024-03-25T19:23:18.096 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.094+0000 7f64aec7b640 1 -- start start 2024-03-25T19:23:18.096 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.094+0000 7f64aec7b640 1 librados: setting wanted keys 2024-03-25T19:23:18.096 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.094+0000 7f64aec7b640 1 librados: calling monclient init 2024-03-25T19:23:18.096 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.094+0000 7f64aec7b640 10 monclient: init 2024-03-25T19:23:18.097 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.095+0000 7f64aec7b640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:18.097 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.095+0000 7f64aec7b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:23:18.097 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.095+0000 7f64aec7b640 10 monclient(hunting): picked mon.noname-b con 0x7f64a80a98f0 addr v1:172.21.15.103:6789/0 2024-03-25T19:23:18.097 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.095+0000 7f64aec7b640 10 monclient(hunting): picked mon.noname-c con 0x7f64a813f710 addr v1:172.21.15.83:6790/0 2024-03-25T19:23:18.097 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.095+0000 7f64aec7b640 10 monclient(hunting): picked mon.noname-a con 0x7f64a81430c0 addr v1:172.21.15.83:6789/0 2024-03-25T19:23:18.097 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.095+0000 7f64aec7b640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64a80abc80 con 0x7f64a81430c0 2024-03-25T19:23:18.097 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.095+0000 7f64aec7b640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64a81d7780 con 0x7f64a80a98f0 2024-03-25T19:23:18.097 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.095+0000 7f64aec7b640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64a81399a0 con 0x7f64a813f710 2024-03-25T19:23:18.098 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.095+0000 7f64aec7b640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:18.098 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.095+0000 7f64aec7b640 10 monclient(hunting): authenticate will time out at 6255.367629s 2024-03-25T19:23:18.105 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.106+0000 7f64ad218640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f64a81430c0 0x7f64a8138600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:43628/0 (socket says 172.21.15.83:43628) 2024-03-25T19:23:18.105 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.106+0000 7f64ad218640 1 -- 172.21.15.83:0/4018356369 learned_addr learned my addr 172.21.15.83:0/4018356369 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:23:18.108 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.109+0000 7f64a57fa640 10 client.?.objecter ms_handle_connect 0x7f64a81430c0 2024-03-25T19:23:18.108 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.109+0000 7f64a57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:18.113 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.113+0000 7f64a57fa640 10 client.?.objecter ms_handle_connect 0x7f64a813f710 2024-03-25T19:23:18.113 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.113+0000 7f64a57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:18.118 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.118+0000 7f64a57fa640 10 client.?.objecter ms_handle_connect 0x7f64a80a98f0 2024-03-25T19:23:18.118 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.118+0000 7f64a57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:18.122 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.122+0000 7f64a57fa640 1 -- 172.21.15.83:0/4018356369 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2102274840 0 0) 0x7f64a81399a0 con 0x7f64a813f710 2024-03-25T19:23:18.122 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.122+0000 7f64a57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:23:18.122 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.122+0000 7f64a57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:18.122 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.122+0000 7f64a57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:18.122 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.122+0000 7f64a57fa640 10 monclient(hunting): my global_id is 4394 2024-03-25T19:23:18.122 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.122+0000 7f64a57fa640 1 -- 172.21.15.83:0/4018356369 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f648c004eb0 con 0x7f64a813f710 2024-03-25T19:23:18.127 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.127+0000 7f64a57fa640 1 -- 172.21.15.83:0/4018356369 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3473912612 0 0) 0x7f64a80abc80 con 0x7f64a81430c0 2024-03-25T19:23:18.127 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.127+0000 7f64a57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:23:18.127 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.127+0000 7f64a57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:18.127 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.127+0000 7f64a57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:18.127 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.127+0000 7f64a57fa640 10 monclient(hunting): my global_id is 4380 2024-03-25T19:23:18.127 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.127+0000 7f64a57fa640 1 -- 172.21.15.83:0/4018356369 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64a81399a0 con 0x7f64a81430c0 2024-03-25T19:23:18.134 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.135+0000 7f64a57fa640 1 -- 172.21.15.83:0/4018356369 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3713403187 0 0) 0x7f64a81d7780 con 0x7f64a80a98f0 2024-03-25T19:23:18.134 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.135+0000 7f64a57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:23:18.134 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.135+0000 7f64a57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:18.134 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.135+0000 7f64a57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:18.135 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.135+0000 7f64a57fa640 10 monclient(hunting): my global_id is 4387 2024-03-25T19:23:18.135 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.135+0000 7f64a57fa640 1 -- 172.21.15.83:0/4018356369 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64a80abc80 con 0x7f64a80a98f0 2024-03-25T19:23:18.143 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.143+0000 7f64a57fa640 1 -- 172.21.15.83:0/4018356369 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2754701630 0 0) 0x7f64a81399a0 con 0x7f64a81430c0 2024-03-25T19:23:18.143 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.143+0000 7f64a57fa640 1 -- 172.21.15.83:0/4018356369 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64a81d7780 con 0x7f64a81430c0 2024-03-25T19:23:18.143 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.143+0000 7f64a57fa640 1 -- 172.21.15.83:0/4018356369 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f64a0001c50 con 0x7f64a81430c0 2024-03-25T19:23:18.143 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.143+0000 7f64a57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:18.143 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.143+0000 7f64a57fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:23:18.143 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.143+0000 7f64a57fa640 10 monclient(hunting): dump: 2024-03-25T19:23:18.143 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:23:18.143 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:23:18.143 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:18.143 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:18.143 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:18.143 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:23:18.144 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:23:18.144 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:23:18.144 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:23:18.144 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:18.144 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.143+0000 7f64a57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:23:18.144 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.144+0000 7f64a57fa640 1 -- 172.21.15.83:0/4018356369 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3216982056 0 0) 0x7f648c004eb0 con 0x7f64a813f710 2024-03-25T19:23:18.144 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.144+0000 7f64a57fa640 1 -- 172.21.15.83:0/4018356369 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64a81399a0 con 0x7f64a813f710 2024-03-25T19:23:18.145 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.144+0000 7f64a57fa640 1 -- 172.21.15.83:0/4018356369 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f6494001b00 con 0x7f64a813f710 2024-03-25T19:23:18.145 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.144+0000 7f64a57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:18.145 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.144+0000 7f64a57fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:23:18.145 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.144+0000 7f64a57fa640 10 monclient(hunting): dump: 2024-03-25T19:23:18.145 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:23:18.145 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:23:18.145 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:18.145 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:18.145 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:18.145 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:23:18.145 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:23:18.145 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:23:18.145 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:23:18.145 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:18.152 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.152+0000 7f64a57fa640 1 -- 172.21.15.83:0/4018356369 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3565996623 0 0) 0x7f64a81399a0 con 0x7f64a813f710 2024-03-25T19:23:18.153 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.152+0000 7f64a57fa640 1 -- 172.21.15.83:0/4018356369 >> v1:172.21.15.103:6789/0 conn(0x7f64a80a98f0 legacy=0x7f64a81d7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:18.153 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.152+0000 7f64a57fa640 1 -- 172.21.15.83:0/4018356369 >> v1:172.21.15.83:6789/0 conn(0x7f64a81430c0 legacy=0x7f64a8138600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:18.153 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.152+0000 7f64a57fa640 10 monclient: _finish_hunting 0 2024-03-25T19:23:18.153 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.152+0000 7f64a57fa640 1 monclient: found mon.c 2024-03-25T19:23:18.153 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.152+0000 7f64a57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:18.153 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.152+0000 7f64a57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:23:18.153 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.152+0000 7f64a57fa640 1 -- 172.21.15.83:0/4018356369 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a81d41e0 con 0x7f64a813f710 2024-03-25T19:23:18.153 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.152+0000 7f64a57fa640 10 monclient: _finish_auth 0 2024-03-25T19:23:18.153 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.152+0000 7f64a57fa640 10 monclient: _check_auth_tickets 2024-03-25T19:23:18.153 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.152+0000 7f64a57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:18.153 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.152+0000 7f64aec7b640 5 monclient: authenticate success, global_id 4394 2024-03-25T19:23:18.153 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.152+0000 7f64aec7b640 10 monclient: _renew_subs 2024-03-25T19:23:18.153 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.152+0000 7f64aec7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:23:18.153 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.152+0000 7f64aec7b640 1 -- 172.21.15.83:0/4018356369 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64a81d8370 con 0x7f64a813f710 2024-03-25T19:23:18.154 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.152+0000 7f64aec7b640 10 client.4394.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:23:18.154 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.152+0000 7f64aec7b640 10 monclient: _renew_subs 2024-03-25T19:23:18.154 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.152+0000 7f64aec7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:23:18.154 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.152+0000 7f64aec7b640 1 -- 172.21.15.83:0/4018356369 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f64a81d5130 con 0x7f64a813f710 2024-03-25T19:23:18.154 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.152+0000 7f64aec7b640 1 librados: init done 2024-03-25T19:23:18.154 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.153+0000 7f64aec7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:18.154 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.153+0000 7f64aec7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:18.154 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.153+0000 7f64aec7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:23:18.155 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.153+0000 7f64aec7b640 1 -- 172.21.15.83:0/4018356369 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64a80627b0 con 0x7f64a813f710 2024-03-25T19:23:18.451 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.452+0000 7f64a57fa640 1 -- 172.21.15.83:0/4018356369 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64940038a0 con 0x7f64a813f710 2024-03-25T19:23:18.452 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.452+0000 7f64a57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:18.452 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.452+0000 7f64a57fa640 1 -- 172.21.15.83:0/4018356369 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f6494003bb0 con 0x7f64a813f710 2024-03-25T19:23:18.452 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.452+0000 7f64a57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:18.452 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.452+0000 7f64a57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:23:18.452 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.452+0000 7f64a57fa640 10 monclient: dump: 2024-03-25T19:23:18.452 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:23:18.452 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:23:18.452 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:18.452 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:18.452 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:18.452 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:23:18.452 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:23:18.452 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:23:18.452 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:23:18.453 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:18.453 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.452+0000 7f64a6ffd640 4 set_mon_vals no callback set 2024-03-25T19:23:18.453 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.453+0000 7f64a57fa640 1 -- 172.21.15.83:0/4018356369 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7f6494019b80 con 0x7f64a813f710 2024-03-25T19:23:18.453 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.453+0000 7f64aca17640 10 monclient: get_auth_request con 0x7f648c06dfb0 auth_method 0 2024-03-25T19:23:18.453 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.453+0000 7f64a57fa640 10 client.4394.objecter ms_handle_connect 0x7f648c06dfb0 2024-03-25T19:23:18.458 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.458+0000 7f64a57fa640 1 -- 172.21.15.83:0/4018356369 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7f6494053680 con 0x7f64a813f710 2024-03-25T19:23:18.458 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.458+0000 7f64a57fa640 10 client.4394.objecter ms_dispatch 0x7f64a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:23:18.458 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.458+0000 7f64a57fa640 3 client.4394.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:23:18.458 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.458+0000 7f64a57fa640 3 client.4394.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:23:18.458 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.458+0000 7f64a57fa640 20 client.4394.objecter dump_active .. 0 homeless 2024-03-25T19:23:18.458 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.458+0000 7f64a57fa640 1 -- 172.21.15.83:0/4018356369 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f649404f190 con 0x7f64a813f710 2024-03-25T19:23:18.458 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.458+0000 7f64a57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:18.458 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.458+0000 7f64a57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:23:18.707 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.707+0000 7f64aec7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:18.707 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.707+0000 7f64aec7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:18.707 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.707+0000 7f64aec7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:23:18.707 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.707+0000 7f64aec7b640 1 -- 172.21.15.83:0/4018356369 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64a813e2b0 con 0x7f64a813f710 2024-03-25T19:23:18.747 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.748+0000 7f64a57fa640 1 -- 172.21.15.83:0/4018356369 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7f64940510c0 con 0x7f64a813f710 2024-03-25T19:23:18.748 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.748+0000 7f64a57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:18.748 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.748+0000 7f64a57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:23:18.748 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:23:18.748 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:23:18.751 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.750+0000 7f64aec7b640 10 librados: watch_flush enter 2024-03-25T19:23:18.751 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.750+0000 7f64aec7b640 10 librados: watch_flush exit 2024-03-25T19:23:18.751 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.750+0000 7f64aec7b640 20 client.4394.objecter shutdown clearing up homeless session... 2024-03-25T19:23:18.751 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.750+0000 7f64aec7b640 10 client.4394.objecter successfully canceled tick 2024-03-25T19:23:18.751 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.751+0000 7f64aec7b640 1 -- 172.21.15.83:0/4018356369 >> v1:172.21.15.103:6800/345072327 conn(0x7f648c06dfb0 legacy=0x7f648c070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:18.751 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.751+0000 7f64aec7b640 10 monclient: shutdown 2024-03-25T19:23:18.751 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.751+0000 7f64aec7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:18.751 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.751+0000 7f64aec7b640 1 -- 172.21.15.83:0/4018356369 >> v1:172.21.15.83:6790/0 conn(0x7f64a813f710 legacy=0x7f64a81d4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:18.751 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.751+0000 7f64aec7b640 1 -- 172.21.15.83:0/4018356369 shutdown_connections 2024-03-25T19:23:18.751 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.751+0000 7f64aec7b640 1 -- 172.21.15.83:0/4018356369 >> 172.21.15.83:0/4018356369 conn(0x7f64a80b7ea0 msgr2=0x7f64a8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:18.751 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.751+0000 7f64aec7b640 1 -- 172.21.15.83:0/4018356369 shutdown_connections 2024-03-25T19:23:18.752 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.751+0000 7f64aec7b640 1 -- 172.21.15.83:0/4018356369 wait complete. 2024-03-25T19:23:18.752 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:18.751+0000 7f64aec7b640 1 librados: shutdown 2024-03-25T19:23:18.764 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:23:22.712 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:04:04.639 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:22.758 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:04:04.685 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:22.790 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:04:04.717 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:22.822 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:04:04.732 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:22.827 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:04:04.737 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:22.846 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:04:04.773 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:22.851 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:04:04.778 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:22.865 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:04:04.775 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:22.882 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:04:04.810 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:25.570 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:23:25.686 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.686+0000 7fea00967640 10 monclient: get_monmap_and_config 2024-03-25T19:23:25.686 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.686+0000 7fea00967640 10 monclient: build_initial_monmap 2024-03-25T19:23:25.686 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.687+0000 7fea00967640 10 monclient: monmap: 2024-03-25T19:23:25.686 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:23:25.686 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:23:25.687 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:23:25.687911+0000 2024-03-25T19:23:25.687 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:23:25.687911+0000 2024-03-25T19:23:25.687 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:25.687 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:23:25.687 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:23:25.687 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:23:25.687 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:23:25.687 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:25.687 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.687+0000 7fea00967640 1 Processor -- start 2024-03-25T19:23:25.688 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.688+0000 7fea00967640 1 -- start start 2024-03-25T19:23:25.688 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.688+0000 7fea00967640 10 monclient: init 2024-03-25T19:23:25.689 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.689+0000 7fea00967640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:25.689 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.689+0000 7fea00967640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:23:25.689 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.689+0000 7fea00967640 10 monclient(hunting): picked mon.noname-b con 0x7fe9fc0a98f0 addr v1:172.21.15.83:6790/0 2024-03-25T19:23:25.689 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.689+0000 7fea00967640 10 monclient(hunting): picked mon.noname-c con 0x7fe9fc13f710 addr v1:172.21.15.103:6789/0 2024-03-25T19:23:25.689 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.689+0000 7fea00967640 10 monclient(hunting): picked mon.noname-a con 0x7fe9fc1430c0 addr v1:172.21.15.83:6789/0 2024-03-25T19:23:25.689 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.689+0000 7fea00967640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9fc142520 con 0x7fe9fc1430c0 2024-03-25T19:23:25.689 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.689+0000 7fea00967640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9fc0b28a0 con 0x7fe9fc13f710 2024-03-25T19:23:25.689 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.689+0000 7fea00967640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9fc0b2a30 con 0x7fe9fc0a98f0 2024-03-25T19:23:25.689 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.689+0000 7fea00967640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:25.689 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.689+0000 7fea00967640 10 monclient(hunting): authenticate will time out at 6262.960896s 2024-03-25T19:23:25.697 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.697+0000 7fe9fa59c640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fe9fc0a98f0 0x7fe9fc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:38106/0 (socket says 172.21.15.83:38106) 2024-03-25T19:23:25.697 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.697+0000 7fe9fa59c640 1 -- 172.21.15.83:0/1474574738 learned_addr learned my addr 172.21.15.83:0/1474574738 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:23:25.725 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.726+0000 7fe9f9d9b640 1 -- 172.21.15.83:0/1474574738 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2152207100 0 0) 0x7fe9fc0b2a30 con 0x7fe9fc0a98f0 2024-03-25T19:23:25.726 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.726+0000 7fe9f9d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:23:25.726 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.726+0000 7fe9f9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:25.726 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.726+0000 7fe9f9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:25.726 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.726+0000 7fe9f9d9b640 10 monclient(hunting): my global_id is 4397 2024-03-25T19:23:25.726 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.726+0000 7fe9f9d9b640 1 -- 172.21.15.83:0/1474574738 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9e8003520 con 0x7fe9fc0a98f0 2024-03-25T19:23:25.731 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.731+0000 7fe9f9d9b640 1 -- 172.21.15.83:0/1474574738 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2478960622 0 0) 0x7fe9fc142520 con 0x7fe9fc1430c0 2024-03-25T19:23:25.731 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.731+0000 7fe9f9d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:23:25.731 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.731+0000 7fe9f9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:25.731 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.731+0000 7fe9f9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:25.731 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.731+0000 7fe9f9d9b640 10 monclient(hunting): my global_id is 4383 2024-03-25T19:23:25.731 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.731+0000 7fe9f9d9b640 1 -- 172.21.15.83:0/1474574738 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9fc0b2a30 con 0x7fe9fc1430c0 2024-03-25T19:23:25.731 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.731+0000 7fe9f9d9b640 1 -- 172.21.15.83:0/1474574738 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2876109423 0 0) 0x7fe9fc0b28a0 con 0x7fe9fc13f710 2024-03-25T19:23:25.731 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.731+0000 7fe9f9d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:23:25.731 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.731+0000 7fe9f9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:25.731 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.731+0000 7fe9f9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:25.731 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.731+0000 7fe9f9d9b640 10 monclient(hunting): my global_id is 4390 2024-03-25T19:23:25.731 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.731+0000 7fe9f9d9b640 1 -- 172.21.15.83:0/1474574738 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9fc142520 con 0x7fe9fc13f710 2024-03-25T19:23:25.743 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.743+0000 7fe9f9d9b640 1 -- 172.21.15.83:0/1474574738 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 876856771 0 0) 0x7fe9fc0b2a30 con 0x7fe9fc1430c0 2024-03-25T19:23:25.743 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.743+0000 7fe9f9d9b640 1 -- 172.21.15.83:0/1474574738 >> v1:172.21.15.83:6790/0 conn(0x7fe9fc0a98f0 legacy=0x7fe9fc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:25.743 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.743+0000 7fe9f9d9b640 1 -- 172.21.15.83:0/1474574738 >> v1:172.21.15.103:6789/0 conn(0x7fe9fc13f710 legacy=0x7fe9fc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:25.743 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.743+0000 7fe9f9d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:23:25.743 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.743+0000 7fe9f9d9b640 1 monclient: found mon.noname-a 2024-03-25T19:23:25.743 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.743+0000 7fe9f9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:25.743 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.743+0000 7fe9f9d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.83:6789/0 2024-03-25T19:23:25.743 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.743+0000 7fe9f9d9b640 1 -- 172.21.15.83:0/1474574738 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9fc0b44c0 con 0x7fe9fc1430c0 2024-03-25T19:23:25.743 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.743+0000 7fe9f9d9b640 10 monclient: _finish_auth 0 2024-03-25T19:23:25.743 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.743+0000 7fe9f9d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:23:25.743 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.743+0000 7fe9f9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:25.744 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.743+0000 7fe9f9d9b640 1 -- 172.21.15.83:0/1474574738 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fe9f40019f0 con 0x7fe9fc1430c0 2024-03-25T19:23:25.744 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.743+0000 7fe9f9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:25.744 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.743+0000 7fe9f9d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:23:25.744 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.743+0000 7fe9f9d9b640 10 monclient: dump: 2024-03-25T19:23:25.744 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:23:25.744 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:23:25.744 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:25.744 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:25.744 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:25.744 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:23:25.744 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:23:25.744 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:23:25.744 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:23:25.744 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:25.744 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.743+0000 7fea00967640 5 monclient: authenticate success, global_id 4383 2024-03-25T19:23:25.744 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.743+0000 7fea00967640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:23:25.753 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.753+0000 7fe9f9d9b640 1 -- 172.21.15.83:0/1474574738 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9f4001b50 con 0x7fe9fc1430c0 2024-03-25T19:23:25.753 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.753+0000 7fe9f9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:25.753 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.753+0000 7fe9f9d9b640 1 -- 172.21.15.83:0/1474574738 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fe9f4001f60 con 0x7fe9fc1430c0 2024-03-25T19:23:25.753 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.753+0000 7fea00967640 10 monclient: get_monmap_and_config success 2024-03-25T19:23:25.753 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.753+0000 7fea00967640 4 set_mon_vals no callback set 2024-03-25T19:23:25.753 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.753+0000 7fe9f9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:25.753 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.753+0000 7fe9f9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:23:25.753 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.753+0000 7fe9f9d9b640 10 monclient: dump: 2024-03-25T19:23:25.753 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:23:25.753 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:23:25.753 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:25.753 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:25.753 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:25.754 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:23:25.754 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:23:25.754 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:23:25.754 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:23:25.754 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:25.755 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.754+0000 7fea00967640 10 monclient: shutdown 2024-03-25T19:23:25.755 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.754+0000 7fea00967640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:25.755 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.754+0000 7fea00967640 1 -- 172.21.15.83:0/1474574738 >> v1:172.21.15.83:6789/0 conn(0x7fe9fc1430c0 legacy=0x7fe9fc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:25.755 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.754+0000 7fea00967640 1 -- 172.21.15.83:0/1474574738 shutdown_connections 2024-03-25T19:23:25.755 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.754+0000 7fea00967640 1 -- 172.21.15.83:0/1474574738 >> 172.21.15.83:0/1474574738 conn(0x7fe9fc0b7ea0 msgr2=0x7fe9fc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:25.755 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.754+0000 7fea00967640 1 -- 172.21.15.83:0/1474574738 shutdown_connections 2024-03-25T19:23:25.755 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.754+0000 7fea00967640 1 -- 172.21.15.83:0/1474574738 wait complete. 2024-03-25T19:23:25.755 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.755+0000 7fea00967640 10 monclient: build_initial_monmap 2024-03-25T19:23:25.755 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.755+0000 7fea00967640 10 monclient: monmap: 2024-03-25T19:23:25.755 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:23:25.755 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:23:25.756 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:23:25.756 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:23:25.756 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:25.756 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:23:25.756 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:23:25.756 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:23:25.756 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:23:25.756 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:25.757 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.756+0000 7fea00967640 1 librados: starting msgr at 2024-03-25T19:23:25.758 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.756+0000 7fea00967640 1 librados: starting objecter 2024-03-25T19:23:25.758 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.756+0000 7fea00967640 1 Processor -- start 2024-03-25T19:23:25.758 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.756+0000 7fea00967640 1 -- start start 2024-03-25T19:23:25.758 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.756+0000 7fea00967640 1 librados: setting wanted keys 2024-03-25T19:23:25.758 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.756+0000 7fea00967640 1 librados: calling monclient init 2024-03-25T19:23:25.758 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.756+0000 7fea00967640 10 monclient: init 2024-03-25T19:23:25.758 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.757+0000 7fea00967640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:25.758 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.757+0000 7fea00967640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:23:25.758 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.757+0000 7fea00967640 10 monclient(hunting): picked mon.noname-b con 0x7fe9fc0a98f0 addr v1:172.21.15.103:6789/0 2024-03-25T19:23:25.758 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.757+0000 7fea00967640 10 monclient(hunting): picked mon.noname-c con 0x7fe9fc13f710 addr v1:172.21.15.83:6790/0 2024-03-25T19:23:25.758 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.757+0000 7fea00967640 10 monclient(hunting): picked mon.noname-a con 0x7fe9fc1430c0 addr v1:172.21.15.83:6789/0 2024-03-25T19:23:25.758 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.757+0000 7fea00967640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9fc139920 con 0x7fe9fc1430c0 2024-03-25T19:23:25.758 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.757+0000 7fea00967640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9fc1db380 con 0x7fe9fc0a98f0 2024-03-25T19:23:25.758 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.757+0000 7fea00967640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9fc1db620 con 0x7fe9fc13f710 2024-03-25T19:23:25.759 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.757+0000 7fea00967640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:25.759 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.757+0000 7fea00967640 10 monclient(hunting): authenticate will time out at 6263.029121s 2024-03-25T19:23:25.771 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.772+0000 7fe9fa59c640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fe9fc0a98f0 0x7fe9fc1dac70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:60274/0 (socket says 172.21.15.83:60274) 2024-03-25T19:23:25.771 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.772+0000 7fe9fa59c640 1 -- 172.21.15.83:0/1589848165 learned_addr learned my addr 172.21.15.83:0/1589848165 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:23:25.774 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.774+0000 7fe9f2ffd640 10 client.?.objecter ms_handle_connect 0x7fe9fc0a98f0 2024-03-25T19:23:25.774 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.774+0000 7fe9f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:25.775 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.775+0000 7fe9f2ffd640 10 client.?.objecter ms_handle_connect 0x7fe9fc1430c0 2024-03-25T19:23:25.775 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.775+0000 7fe9f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:25.779 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.779+0000 7fe9f2ffd640 10 client.?.objecter ms_handle_connect 0x7fe9fc13f710 2024-03-25T19:23:25.779 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.779+0000 7fe9f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:25.786 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.787+0000 7fe9f2ffd640 1 -- 172.21.15.83:0/1589848165 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3809360829 0 0) 0x7fe9fc1db380 con 0x7fe9fc0a98f0 2024-03-25T19:23:25.786 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.787+0000 7fe9f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:23:25.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.787+0000 7fe9f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:25.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.787+0000 7fe9f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:25.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.787+0000 7fe9f2ffd640 10 monclient(hunting): my global_id is 4393 2024-03-25T19:23:25.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.787+0000 7fe9f2ffd640 1 -- 172.21.15.83:0/1589848165 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9dc004eb0 con 0x7fe9fc0a98f0 2024-03-25T19:23:25.795 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.796+0000 7fe9f2ffd640 1 -- 172.21.15.83:0/1589848165 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 66182989 0 0) 0x7fe9fc139920 con 0x7fe9fc1430c0 2024-03-25T19:23:25.795 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.796+0000 7fe9f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:23:25.795 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.796+0000 7fe9f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:25.795 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.796+0000 7fe9f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:25.795 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.796+0000 7fe9f2ffd640 10 monclient(hunting): my global_id is 4386 2024-03-25T19:23:25.796 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.796+0000 7fe9f2ffd640 1 -- 172.21.15.83:0/1589848165 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9fc1db380 con 0x7fe9fc1430c0 2024-03-25T19:23:25.796 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.797+0000 7fe9f2ffd640 1 -- 172.21.15.83:0/1589848165 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3409447044 0 0) 0x7fe9fc1db620 con 0x7fe9fc13f710 2024-03-25T19:23:25.796 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.797+0000 7fe9f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:23:25.796 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.797+0000 7fe9f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:25.796 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.797+0000 7fe9f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:25.797 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.797+0000 7fe9f2ffd640 10 monclient(hunting): my global_id is 4400 2024-03-25T19:23:25.797 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.797+0000 7fe9f2ffd640 1 -- 172.21.15.83:0/1589848165 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9fc139920 con 0x7fe9fc13f710 2024-03-25T19:23:25.808 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.809+0000 7fe9f2ffd640 1 -- 172.21.15.83:0/1589848165 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2625520967 0 0) 0x7fe9fc1db380 con 0x7fe9fc1430c0 2024-03-25T19:23:25.808 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.809+0000 7fe9f2ffd640 1 -- 172.21.15.83:0/1589848165 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9fc1db620 con 0x7fe9fc1430c0 2024-03-25T19:23:25.809 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.809+0000 7fe9f2ffd640 1 -- 172.21.15.83:0/1589848165 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fe9f4001e60 con 0x7fe9fc1430c0 2024-03-25T19:23:25.809 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.809+0000 7fe9f2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:25.809 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.809+0000 7fe9f2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:23:25.809 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.809+0000 7fe9f2ffd640 10 monclient(hunting): dump: 2024-03-25T19:23:25.809 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:23:25.809 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:23:25.809 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:25.809 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:25.809 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:25.809 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:23:25.809 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:23:25.809 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:23:25.809 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:23:25.809 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:25.809 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.809+0000 7fe9f2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:23:25.813 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.813+0000 7fe9f2ffd640 1 -- 172.21.15.83:0/1589848165 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 15761309 0 0) 0x7fe9dc004eb0 con 0x7fe9fc0a98f0 2024-03-25T19:23:25.813 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.813+0000 7fe9f2ffd640 1 -- 172.21.15.83:0/1589848165 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9fc1db380 con 0x7fe9fc0a98f0 2024-03-25T19:23:25.813 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.813+0000 7fe9f2ffd640 1 -- 172.21.15.83:0/1589848165 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fe9ec001950 con 0x7fe9fc0a98f0 2024-03-25T19:23:25.813 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.813+0000 7fe9f2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:25.813 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.813+0000 7fe9f2ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:23:25.813 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.813+0000 7fe9f2ffd640 10 monclient(hunting): dump: 2024-03-25T19:23:25.813 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:23:25.813 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:23:25.813 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:25.813 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:25.813 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:25.813 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:23:25.813 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:23:25.814 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:23:25.814 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:23:25.814 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:25.823 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.824+0000 7fe9f2ffd640 1 -- 172.21.15.83:0/1589848165 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 501900283 0 0) 0x7fe9fc139920 con 0x7fe9fc13f710 2024-03-25T19:23:25.823 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.824+0000 7fe9f2ffd640 1 -- 172.21.15.83:0/1589848165 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9dc004eb0 con 0x7fe9fc13f710 2024-03-25T19:23:25.824 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.824+0000 7fe9f2ffd640 1 -- 172.21.15.83:0/1589848165 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fe9e4001830 con 0x7fe9fc13f710 2024-03-25T19:23:25.824 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.824+0000 7fe9f2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:25.824 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.824+0000 7fe9f2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:23:25.824 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.824+0000 7fe9f2ffd640 10 monclient(hunting): dump: 2024-03-25T19:23:25.824 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:23:25.824 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:23:25.824 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:25.824 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:25.824 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:25.824 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:23:25.824 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:23:25.824 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:23:25.824 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:23:25.824 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:25.834 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.834+0000 7fe9f2ffd640 1 -- 172.21.15.83:0/1589848165 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 652872819 0 0) 0x7fe9fc1db620 con 0x7fe9fc1430c0 2024-03-25T19:23:25.834 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.834+0000 7fe9f2ffd640 1 -- 172.21.15.83:0/1589848165 >> v1:172.21.15.83:6790/0 conn(0x7fe9fc13f710 legacy=0x7fe9fc0b0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:25.834 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.834+0000 7fe9f2ffd640 1 -- 172.21.15.83:0/1589848165 >> v1:172.21.15.103:6789/0 conn(0x7fe9fc0a98f0 legacy=0x7fe9fc1dac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:25.834 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.834+0000 7fe9f2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:23:25.834 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.834+0000 7fe9f2ffd640 1 monclient: found mon.a 2024-03-25T19:23:25.834 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.834+0000 7fe9f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:25.834 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.834+0000 7fe9f2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:23:25.835 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.834+0000 7fe9f2ffd640 1 -- 172.21.15.83:0/1589848165 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9fc0b0dc0 con 0x7fe9fc1430c0 2024-03-25T19:23:25.835 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.834+0000 7fe9f2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:23:25.835 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.834+0000 7fe9f2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:23:25.835 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.834+0000 7fe9f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:25.835 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.834+0000 7fea00967640 5 monclient: authenticate success, global_id 4386 2024-03-25T19:23:25.835 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.834+0000 7fea00967640 10 monclient: _renew_subs 2024-03-25T19:23:25.835 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.834+0000 7fea00967640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:23:25.835 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.834+0000 7fea00967640 1 -- 172.21.15.83:0/1589848165 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9fc0b1010 con 0x7fe9fc1430c0 2024-03-25T19:23:25.835 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.834+0000 7fea00967640 10 client.4386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:23:25.835 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.834+0000 7fea00967640 10 monclient: _renew_subs 2024-03-25T19:23:25.835 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.834+0000 7fea00967640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:23:25.835 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.834+0000 7fea00967640 1 -- 172.21.15.83:0/1589848165 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9fc0b0910 con 0x7fe9fc1430c0 2024-03-25T19:23:25.835 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.834+0000 7fea00967640 1 librados: init done 2024-03-25T19:23:25.836 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.835+0000 7fea00967640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:25.836 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.835+0000 7fea00967640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:25.836 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.835+0000 7fea00967640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:23:25.836 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:25.835+0000 7fea00967640 1 -- 172.21.15.83:0/1589848165 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9fc0b0c10 con 0x7fe9fc1430c0 2024-03-25T19:23:26.163 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:26.163+0000 7fe9f2ffd640 1 -- 172.21.15.83:0/1589848165 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9f4002020 con 0x7fe9fc1430c0 2024-03-25T19:23:26.163 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:26.163+0000 7fe9f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:26.163 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:26.163+0000 7fe9f2ffd640 1 -- 172.21.15.83:0/1589848165 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fe9f4003dc0 con 0x7fe9fc1430c0 2024-03-25T19:23:26.163 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:26.163+0000 7fe9f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:26.163 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:26.163+0000 7fe9f2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:23:26.163 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:26.163+0000 7fe9f2ffd640 10 monclient: dump: 2024-03-25T19:23:26.163 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:23:26.163 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:23:26.163 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:26.163 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:26.163 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:26.164 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:23:26.164 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:23:26.164 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:23:26.164 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:23:26.164 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:26.164 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:26.163+0000 7fe9f959a640 4 set_mon_vals no callback set 2024-03-25T19:23:26.164 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:26.164+0000 7fe9f2ffd640 1 -- 172.21.15.83:0/1589848165 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7fe9f4019d90 con 0x7fe9fc1430c0 2024-03-25T19:23:26.165 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:26.165+0000 7fe9fa59c640 10 monclient: get_auth_request con 0x7fe9dc06e0b0 auth_method 0 2024-03-25T19:23:26.165 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:26.165+0000 7fe9f2ffd640 10 client.4386.objecter ms_handle_connect 0x7fe9dc06e0b0 2024-03-25T19:23:26.171 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:26.171+0000 7fe9f2ffd640 1 -- 172.21.15.83:0/1589848165 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7fe9f40538e0 con 0x7fe9fc1430c0 2024-03-25T19:23:26.171 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:26.171+0000 7fe9f2ffd640 10 client.4386.objecter ms_dispatch 0x7fe9fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:23:26.171 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:26.171+0000 7fe9f2ffd640 3 client.4386.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:23:26.171 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:26.171+0000 7fe9f2ffd640 3 client.4386.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:23:26.172 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:26.171+0000 7fe9f2ffd640 20 client.4386.objecter dump_active .. 0 homeless 2024-03-25T19:23:26.172 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:26.171+0000 7fe9f2ffd640 1 -- 172.21.15.83:0/1589848165 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe9f404f3f0 con 0x7fe9fc1430c0 2024-03-25T19:23:26.172 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:26.172+0000 7fe9f2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:26.172 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:26.172+0000 7fe9f2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:23:26.386 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:26.386+0000 7fea00967640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:26.386 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:26.386+0000 7fea00967640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:26.386 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:26.386+0000 7fea00967640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:23:26.386 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:26.386+0000 7fea00967640 1 -- 172.21.15.83:0/1589848165 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9fc1db7b0 con 0x7fe9fc1430c0 2024-03-25T19:23:26.421 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:26.421+0000 7fe9f2ffd640 1 -- 172.21.15.83:0/1589848165 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7fe9f4051320 con 0x7fe9fc1430c0 2024-03-25T19:23:26.421 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:26.422+0000 7fe9f2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:26.422 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:26.422+0000 7fe9f2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:23:26.422 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:23:26.422 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:23:26.425 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:26.424+0000 7fea00967640 10 librados: watch_flush enter 2024-03-25T19:23:26.425 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:26.424+0000 7fea00967640 10 librados: watch_flush exit 2024-03-25T19:23:26.425 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:26.424+0000 7fea00967640 20 client.4386.objecter shutdown clearing up homeless session... 2024-03-25T19:23:26.425 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:26.424+0000 7fea00967640 10 client.4386.objecter successfully canceled tick 2024-03-25T19:23:26.425 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:26.425+0000 7fea00967640 1 -- 172.21.15.83:0/1589848165 >> v1:172.21.15.103:6800/345072327 conn(0x7fe9dc06e0b0 legacy=0x7fe9dc070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:26.425 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:26.425+0000 7fea00967640 10 monclient: shutdown 2024-03-25T19:23:26.425 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:26.425+0000 7fea00967640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:26.425 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:26.425+0000 7fea00967640 1 -- 172.21.15.83:0/1589848165 >> v1:172.21.15.83:6789/0 conn(0x7fe9fc1430c0 legacy=0x7fe9fc137610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:26.425 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:26.425+0000 7fea00967640 1 -- 172.21.15.83:0/1589848165 shutdown_connections 2024-03-25T19:23:26.425 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:26.425+0000 7fea00967640 1 -- 172.21.15.83:0/1589848165 >> 172.21.15.83:0/1589848165 conn(0x7fe9fc0b7ea0 msgr2=0x7fe9fc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:26.425 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:26.425+0000 7fea00967640 1 -- 172.21.15.83:0/1589848165 shutdown_connections 2024-03-25T19:23:26.425 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:26.425+0000 7fea00967640 1 -- 172.21.15.83:0/1589848165 wait complete. 2024-03-25T19:23:26.425 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:26.425+0000 7fea00967640 1 librados: shutdown 2024-03-25T19:23:26.437 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:23:33.243 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:23:33.360 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.359+0000 7ff40dcfb640 10 monclient: get_monmap_and_config 2024-03-25T19:23:33.360 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.359+0000 7ff40dcfb640 10 monclient: build_initial_monmap 2024-03-25T19:23:33.360 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.359+0000 7ff40dcfb640 10 monclient: monmap: 2024-03-25T19:23:33.360 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:23:33.360 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:23:33.360 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:23:33.361361+0000 2024-03-25T19:23:33.360 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:23:33.361361+0000 2024-03-25T19:23:33.360 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:33.360 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:23:33.360 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:23:33.360 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:23:33.360 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:23:33.360 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:33.362 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.360+0000 7ff40dcfb640 1 Processor -- start 2024-03-25T19:23:33.362 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.360+0000 7ff40dcfb640 1 -- start start 2024-03-25T19:23:33.362 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.360+0000 7ff40dcfb640 10 monclient: init 2024-03-25T19:23:33.362 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.360+0000 7ff40dcfb640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:33.362 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.360+0000 7ff40dcfb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:23:33.362 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.360+0000 7ff40dcfb640 10 monclient(hunting): picked mon.noname-a con 0x7ff4080b4d60 addr v1:172.21.15.83:6789/0 2024-03-25T19:23:33.362 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.360+0000 7ff40dcfb640 10 monclient(hunting): picked mon.noname-b con 0x7ff4080b1730 addr v1:172.21.15.83:6790/0 2024-03-25T19:23:33.362 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.360+0000 7ff40dcfb640 10 monclient(hunting): picked mon.noname-c con 0x7ff40814be90 addr v1:172.21.15.103:6789/0 2024-03-25T19:23:33.362 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.360+0000 7ff40dcfb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4080af940 con 0x7ff4080b4d60 2024-03-25T19:23:33.362 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.360+0000 7ff40dcfb640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4080b0a70 con 0x7ff40814be90 2024-03-25T19:23:33.362 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.360+0000 7ff40dcfb640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4080b0d50 con 0x7ff4080b1730 2024-03-25T19:23:33.363 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.361+0000 7ff40dcfb640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:33.363 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.361+0000 7ff40dcfb640 10 monclient(hunting): authenticate will time out at 6270.633755s 2024-03-25T19:23:33.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.370+0000 7ff407fff640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7ff40814be90 0x7ff40814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:36734/0 (socket says 172.21.15.83:36734) 2024-03-25T19:23:33.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.370+0000 7ff407fff640 1 -- 172.21.15.83:0/1504530576 learned_addr learned my addr 172.21.15.83:0/1504530576 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:23:33.392 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.392+0000 7ff4067fc640 1 -- 172.21.15.83:0/1504530576 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4002763971 0 0) 0x7ff4080af940 con 0x7ff4080b4d60 2024-03-25T19:23:33.392 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.392+0000 7ff4067fc640 10 monclient(hunting): _negotiate 2024-03-25T19:23:33.392 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.392+0000 7ff4067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:33.393 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.392+0000 7ff4067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:33.393 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.392+0000 7ff4067fc640 10 monclient(hunting): my global_id is 4389 2024-03-25T19:23:33.393 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.392+0000 7ff4067fc640 1 -- 172.21.15.83:0/1504530576 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3f0003520 con 0x7ff4080b4d60 2024-03-25T19:23:33.395 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.394+0000 7ff4067fc640 1 -- 172.21.15.83:0/1504530576 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3978797538 0 0) 0x7ff4080b0a70 con 0x7ff40814be90 2024-03-25T19:23:33.395 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.395+0000 7ff4067fc640 10 monclient(hunting): _negotiate 2024-03-25T19:23:33.395 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.395+0000 7ff4067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:33.395 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.395+0000 7ff4067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:33.396 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.395+0000 7ff4067fc640 10 monclient(hunting): my global_id is 4396 2024-03-25T19:23:33.396 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.395+0000 7ff4067fc640 1 -- 172.21.15.83:0/1504530576 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4080af940 con 0x7ff40814be90 2024-03-25T19:23:33.399 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.398+0000 7ff4067fc640 1 -- 172.21.15.83:0/1504530576 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 156369647 0 0) 0x7ff4080b0d50 con 0x7ff4080b1730 2024-03-25T19:23:33.399 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.398+0000 7ff4067fc640 10 monclient(hunting): _negotiate 2024-03-25T19:23:33.399 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.398+0000 7ff4067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:33.399 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.398+0000 7ff4067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:33.399 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.398+0000 7ff4067fc640 10 monclient(hunting): my global_id is 4403 2024-03-25T19:23:33.399 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.398+0000 7ff4067fc640 1 -- 172.21.15.83:0/1504530576 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4080b0a70 con 0x7ff4080b1730 2024-03-25T19:23:33.408 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.407+0000 7ff4067fc640 1 -- 172.21.15.83:0/1504530576 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1541179887 0 0) 0x7ff4080af940 con 0x7ff40814be90 2024-03-25T19:23:33.408 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.407+0000 7ff4067fc640 1 -- 172.21.15.83:0/1504530576 >> v1:172.21.15.83:6790/0 conn(0x7ff4080b1730 legacy=0x7ff4080b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:33.408 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.407+0000 7ff4067fc640 1 -- 172.21.15.83:0/1504530576 >> v1:172.21.15.83:6789/0 conn(0x7ff4080b4d60 legacy=0x7ff40813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:33.408 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.407+0000 7ff4067fc640 10 monclient: _finish_hunting 0 2024-03-25T19:23:33.408 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.407+0000 7ff4067fc640 1 monclient: found mon.noname-c 2024-03-25T19:23:33.408 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.407+0000 7ff4067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:33.408 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.407+0000 7ff4067fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.103:6789/0 2024-03-25T19:23:33.408 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.407+0000 7ff4067fc640 1 -- 172.21.15.83:0/1504530576 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4080b1030 con 0x7ff40814be90 2024-03-25T19:23:33.408 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.407+0000 7ff4067fc640 10 monclient: _finish_auth 0 2024-03-25T19:23:33.408 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.407+0000 7ff4067fc640 10 monclient: _check_auth_tickets 2024-03-25T19:23:33.408 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.407+0000 7ff4067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:33.408 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.407+0000 7ff4067fc640 1 -- 172.21.15.83:0/1504530576 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7ff4000019f0 con 0x7ff40814be90 2024-03-25T19:23:33.409 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.407+0000 7ff4067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:33.409 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.407+0000 7ff4067fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:23:33.409 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.407+0000 7ff4067fc640 10 monclient: dump: 2024-03-25T19:23:33.409 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:23:33.409 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:23:33.409 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:33.409 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:33.409 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:33.409 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:23:33.409 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:23:33.409 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:23:33.409 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:23:33.409 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:33.409 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.407+0000 7ff40dcfb640 5 monclient: authenticate success, global_id 4396 2024-03-25T19:23:33.409 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.407+0000 7ff40dcfb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:23:33.419 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.418+0000 7ff4067fc640 1 -- 172.21.15.83:0/1504530576 <== mon.1 v1:172.21.15.103:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff400001b50 con 0x7ff40814be90 2024-03-25T19:23:33.419 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.418+0000 7ff4067fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:33.419 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.418+0000 7ff4067fc640 1 -- 172.21.15.83:0/1504530576 <== mon.1 v1:172.21.15.103:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7ff400001f60 con 0x7ff40814be90 2024-03-25T19:23:33.419 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.418+0000 7ff40dcfb640 10 monclient: get_monmap_and_config success 2024-03-25T19:23:33.419 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.418+0000 7ff40dcfb640 4 set_mon_vals no callback set 2024-03-25T19:23:33.419 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.418+0000 7ff4067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:33.419 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.418+0000 7ff4067fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:23:33.419 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.418+0000 7ff4067fc640 10 monclient: dump: 2024-03-25T19:23:33.419 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:23:33.419 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:23:33.420 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:33.420 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:33.420 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:33.420 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:23:33.420 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:23:33.420 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:23:33.420 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:23:33.420 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:33.421 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.419+0000 7ff40dcfb640 10 monclient: shutdown 2024-03-25T19:23:33.421 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.419+0000 7ff40dcfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:33.421 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.419+0000 7ff40dcfb640 1 -- 172.21.15.83:0/1504530576 >> v1:172.21.15.103:6789/0 conn(0x7ff40814be90 legacy=0x7ff40814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:33.422 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.419+0000 7ff40dcfb640 1 -- 172.21.15.83:0/1504530576 shutdown_connections 2024-03-25T19:23:33.422 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.419+0000 7ff40dcfb640 1 -- 172.21.15.83:0/1504530576 >> 172.21.15.83:0/1504530576 conn(0x7ff4080617a0 msgr2=0x7ff408061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:33.422 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.419+0000 7ff40dcfb640 1 -- 172.21.15.83:0/1504530576 shutdown_connections 2024-03-25T19:23:33.422 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.419+0000 7ff40dcfb640 1 -- 172.21.15.83:0/1504530576 wait complete. 2024-03-25T19:23:33.422 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.420+0000 7ff40dcfb640 10 monclient: build_initial_monmap 2024-03-25T19:23:33.422 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.420+0000 7ff40dcfb640 10 monclient: monmap: 2024-03-25T19:23:33.422 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:23:33.422 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:23:33.422 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:23:33.422 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:23:33.422 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:33.422 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:23:33.422 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:23:33.422 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:23:33.422 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:23:33.423 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:33.423 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.420+0000 7ff40dcfb640 1 librados: starting msgr at 2024-03-25T19:23:33.423 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.420+0000 7ff40dcfb640 1 librados: starting objecter 2024-03-25T19:23:33.423 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.420+0000 7ff40dcfb640 1 Processor -- start 2024-03-25T19:23:33.423 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.421+0000 7ff40dcfb640 1 -- start start 2024-03-25T19:23:33.423 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.421+0000 7ff40dcfb640 1 librados: setting wanted keys 2024-03-25T19:23:33.423 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.421+0000 7ff40dcfb640 1 librados: calling monclient init 2024-03-25T19:23:33.423 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.421+0000 7ff40dcfb640 10 monclient: init 2024-03-25T19:23:33.424 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.421+0000 7ff40dcfb640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:33.424 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.421+0000 7ff40dcfb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:23:33.424 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.422+0000 7ff40dcfb640 10 monclient(hunting): picked mon.noname-a con 0x7ff4080b1730 addr v1:172.21.15.83:6789/0 2024-03-25T19:23:33.424 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.422+0000 7ff40dcfb640 10 monclient(hunting): picked mon.noname-c con 0x7ff4080b4d60 addr v1:172.21.15.83:6790/0 2024-03-25T19:23:33.424 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.422+0000 7ff40dcfb640 10 monclient(hunting): picked mon.noname-b con 0x7ff40814be90 addr v1:172.21.15.103:6789/0 2024-03-25T19:23:33.424 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.422+0000 7ff40dcfb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4080aa900 con 0x7ff4080b1730 2024-03-25T19:23:33.424 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.422+0000 7ff40dcfb640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4080a9580 con 0x7ff40814be90 2024-03-25T19:23:33.425 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.422+0000 7ff40dcfb640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff408142f50 con 0x7ff4080b4d60 2024-03-25T19:23:33.425 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.422+0000 7ff40dcfb640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:33.425 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.422+0000 7ff40dcfb640 10 monclient(hunting): authenticate will time out at 6270.694979s 2024-03-25T19:23:33.441 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.440+0000 7ff407fff640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7ff40814be90 0x7ff4081d8120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:36774/0 (socket says 172.21.15.83:36774) 2024-03-25T19:23:33.441 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.440+0000 7ff407fff640 1 -- 172.21.15.83:0/2684327367 learned_addr learned my addr 172.21.15.83:0/2684327367 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:23:33.444 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.444+0000 7ff3effff640 10 client.?.objecter ms_handle_connect 0x7ff40814be90 2024-03-25T19:23:33.445 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.444+0000 7ff3effff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:33.448 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.448+0000 7ff3effff640 10 client.?.objecter ms_handle_connect 0x7ff4080b1730 2024-03-25T19:23:33.448 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.448+0000 7ff3effff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:33.449 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.449+0000 7ff3effff640 10 client.?.objecter ms_handle_connect 0x7ff4080b4d60 2024-03-25T19:23:33.450 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.449+0000 7ff3effff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:33.454 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.453+0000 7ff3effff640 1 -- 172.21.15.83:0/2684327367 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3682023626 0 0) 0x7ff4080a9580 con 0x7ff40814be90 2024-03-25T19:23:33.454 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.453+0000 7ff3effff640 10 monclient(hunting): _negotiate 2024-03-25T19:23:33.454 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.453+0000 7ff3effff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:33.454 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.453+0000 7ff3effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:33.454 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.453+0000 7ff3effff640 10 monclient(hunting): my global_id is 4399 2024-03-25T19:23:33.454 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.453+0000 7ff3effff640 1 -- 172.21.15.83:0/2684327367 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3f4004eb0 con 0x7ff40814be90 2024-03-25T19:23:33.455 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.455+0000 7ff3effff640 1 -- 172.21.15.83:0/2684327367 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1523639821 0 0) 0x7ff408142f50 con 0x7ff4080b4d60 2024-03-25T19:23:33.456 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.455+0000 7ff3effff640 10 monclient(hunting): _negotiate 2024-03-25T19:23:33.456 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.455+0000 7ff3effff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:33.456 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.455+0000 7ff3effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:33.456 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.455+0000 7ff3effff640 10 monclient(hunting): my global_id is 4406 2024-03-25T19:23:33.456 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.455+0000 7ff3effff640 1 -- 172.21.15.83:0/2684327367 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4080a9580 con 0x7ff4080b4d60 2024-03-25T19:23:33.459 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.458+0000 7ff3effff640 1 -- 172.21.15.83:0/2684327367 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1583023309 0 0) 0x7ff4080aa900 con 0x7ff4080b1730 2024-03-25T19:23:33.459 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.458+0000 7ff3effff640 10 monclient(hunting): _negotiate 2024-03-25T19:23:33.459 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.458+0000 7ff3effff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:33.459 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.458+0000 7ff3effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:33.459 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.458+0000 7ff3effff640 10 monclient(hunting): my global_id is 4392 2024-03-25T19:23:33.459 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.458+0000 7ff3effff640 1 -- 172.21.15.83:0/2684327367 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff408142f50 con 0x7ff4080b1730 2024-03-25T19:23:33.467 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.466+0000 7ff3effff640 1 -- 172.21.15.83:0/2684327367 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2869449299 0 0) 0x7ff3f4004eb0 con 0x7ff40814be90 2024-03-25T19:23:33.467 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.466+0000 7ff3effff640 1 -- 172.21.15.83:0/2684327367 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff4080aa900 con 0x7ff40814be90 2024-03-25T19:23:33.467 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.466+0000 7ff3effff640 1 -- 172.21.15.83:0/2684327367 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7ff400001f60 con 0x7ff40814be90 2024-03-25T19:23:33.467 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.466+0000 7ff3effff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:33.467 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.466+0000 7ff3effff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:23:33.467 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.466+0000 7ff3effff640 10 monclient(hunting): dump: 2024-03-25T19:23:33.467 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:23:33.467 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:23:33.467 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:33.467 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:33.467 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:33.467 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:23:33.467 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:23:33.467 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:23:33.467 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:23:33.468 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:33.468 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.466+0000 7ff3effff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:23:33.468 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.467+0000 7ff3effff640 1 -- 172.21.15.83:0/2684327367 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3994722869 0 0) 0x7ff408142f50 con 0x7ff4080b1730 2024-03-25T19:23:33.468 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.467+0000 7ff3effff640 1 -- 172.21.15.83:0/2684327367 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3f4004eb0 con 0x7ff4080b1730 2024-03-25T19:23:33.468 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.468+0000 7ff3effff640 1 -- 172.21.15.83:0/2684327367 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7ff3f80017b0 con 0x7ff4080b1730 2024-03-25T19:23:33.468 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.468+0000 7ff3effff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:33.469 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.468+0000 7ff3effff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:23:33.469 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.468+0000 7ff3effff640 10 monclient(hunting): dump: 2024-03-25T19:23:33.469 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:23:33.469 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:23:33.469 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:33.469 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:33.469 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:33.469 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:23:33.469 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:23:33.469 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:23:33.469 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:23:33.469 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:33.481 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.480+0000 7ff3effff640 1 -- 172.21.15.83:0/2684327367 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2415826687 0 0) 0x7ff3f4004eb0 con 0x7ff4080b1730 2024-03-25T19:23:33.481 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.480+0000 7ff3effff640 1 -- 172.21.15.83:0/2684327367 >> v1:172.21.15.83:6790/0 conn(0x7ff4080b4d60 legacy=0x7ff4081d7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:33.481 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.480+0000 7ff3effff640 1 -- 172.21.15.83:0/2684327367 >> v1:172.21.15.103:6789/0 conn(0x7ff40814be90 legacy=0x7ff4081d8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:33.481 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.480+0000 7ff3effff640 10 monclient: _finish_hunting 0 2024-03-25T19:23:33.481 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.480+0000 7ff3effff640 1 monclient: found mon.a 2024-03-25T19:23:33.481 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.480+0000 7ff3effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:33.481 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.480+0000 7ff3effff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:23:33.481 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.480+0000 7ff3effff640 1 -- 172.21.15.83:0/2684327367 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4081430e0 con 0x7ff4080b1730 2024-03-25T19:23:33.482 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.480+0000 7ff3effff640 10 monclient: _finish_auth 0 2024-03-25T19:23:33.482 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.480+0000 7ff3effff640 10 monclient: _check_auth_tickets 2024-03-25T19:23:33.482 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.480+0000 7ff3effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:33.482 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.480+0000 7ff40dcfb640 5 monclient: authenticate success, global_id 4392 2024-03-25T19:23:33.482 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.480+0000 7ff40dcfb640 10 monclient: _renew_subs 2024-03-25T19:23:33.482 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.480+0000 7ff40dcfb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:23:33.482 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.480+0000 7ff40dcfb640 1 -- 172.21.15.83:0/2684327367 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff408140380 con 0x7ff4080b1730 2024-03-25T19:23:33.482 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.480+0000 7ff40dcfb640 10 client.4392.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:23:33.482 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.480+0000 7ff40dcfb640 10 monclient: _renew_subs 2024-03-25T19:23:33.482 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.480+0000 7ff40dcfb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:23:33.482 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.480+0000 7ff40dcfb640 1 -- 172.21.15.83:0/2684327367 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff408142970 con 0x7ff4080b1730 2024-03-25T19:23:33.482 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.480+0000 7ff40dcfb640 1 librados: init done 2024-03-25T19:23:33.483 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.481+0000 7ff40dcfb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:33.483 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.481+0000 7ff40dcfb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:33.483 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.481+0000 7ff40dcfb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:23:33.483 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.481+0000 7ff40dcfb640 1 -- 172.21.15.83:0/2684327367 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4080a9710 con 0x7ff4080b1730 2024-03-25T19:23:33.788 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.787+0000 7ff3effff640 1 -- 172.21.15.83:0/2684327367 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3f8001cc0 con 0x7ff4080b1730 2024-03-25T19:23:33.788 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.787+0000 7ff3effff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:33.788 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.787+0000 7ff3effff640 1 -- 172.21.15.83:0/2684327367 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7ff3f80037b0 con 0x7ff4080b1730 2024-03-25T19:23:33.788 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.787+0000 7ff3effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:33.788 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.787+0000 7ff3effff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:23:33.788 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.787+0000 7ff3effff640 10 monclient: dump: 2024-03-25T19:23:33.788 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:23:33.788 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:23:33.788 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:33.789 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:33.789 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:33.789 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:23:33.789 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:23:33.789 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:23:33.789 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:23:33.789 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:33.789 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.787+0000 7ff405ffb640 4 set_mon_vals no callback set 2024-03-25T19:23:33.790 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.788+0000 7ff3effff640 1 -- 172.21.15.83:0/2684327367 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7ff3f8019780 con 0x7ff4080b1730 2024-03-25T19:23:33.790 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.788+0000 7ff406ffd640 10 monclient: get_auth_request con 0x7ff3f406dfb0 auth_method 0 2024-03-25T19:23:33.790 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.789+0000 7ff3effff640 10 client.4392.objecter ms_handle_connect 0x7ff3f406dfb0 2024-03-25T19:23:33.795 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.794+0000 7ff3effff640 1 -- 172.21.15.83:0/2684327367 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7ff3f80534e0 con 0x7ff4080b1730 2024-03-25T19:23:33.795 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.794+0000 7ff3effff640 10 client.4392.objecter ms_dispatch 0x7ff408000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:23:33.795 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.794+0000 7ff3effff640 3 client.4392.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:23:33.795 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.794+0000 7ff3effff640 3 client.4392.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:23:33.795 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.794+0000 7ff3effff640 20 client.4392.objecter dump_active .. 0 homeless 2024-03-25T19:23:33.795 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.794+0000 7ff3effff640 1 -- 172.21.15.83:0/2684327367 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff3f804ee20 con 0x7ff4080b1730 2024-03-25T19:23:33.795 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.794+0000 7ff3effff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:33.795 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:33.794+0000 7ff3effff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:23:34.048 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:34.047+0000 7ff40dcfb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:34.048 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:34.047+0000 7ff40dcfb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:34.048 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:34.047+0000 7ff40dcfb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:23:34.048 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:34.047+0000 7ff40dcfb640 1 -- 172.21.15.83:0/2684327367 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff4080620b0 con 0x7ff4080b1730 2024-03-25T19:23:34.073 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:34.072+0000 7ff3effff640 1 -- 172.21.15.83:0/2684327367 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7ff3f8050f20 con 0x7ff4080b1730 2024-03-25T19:23:34.073 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:34.072+0000 7ff3effff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:34.073 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:34.072+0000 7ff3effff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:23:34.073 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:23:34.073 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:23:34.076 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:34.075+0000 7ff40dcfb640 10 librados: watch_flush enter 2024-03-25T19:23:34.076 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:34.075+0000 7ff40dcfb640 10 librados: watch_flush exit 2024-03-25T19:23:34.076 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:34.075+0000 7ff40dcfb640 20 client.4392.objecter shutdown clearing up homeless session... 2024-03-25T19:23:34.076 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:34.075+0000 7ff40dcfb640 10 client.4392.objecter successfully canceled tick 2024-03-25T19:23:34.076 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:34.075+0000 7ff40dcfb640 1 -- 172.21.15.83:0/2684327367 >> v1:172.21.15.103:6800/345072327 conn(0x7ff3f406dfb0 legacy=0x7ff3f4070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:34.077 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:34.075+0000 7ff40dcfb640 10 monclient: shutdown 2024-03-25T19:23:34.077 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:34.075+0000 7ff40dcfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:34.077 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:34.075+0000 7ff40dcfb640 1 -- 172.21.15.83:0/2684327367 >> v1:172.21.15.83:6789/0 conn(0x7ff4080b1730 legacy=0x7ff408137e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:34.077 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:34.075+0000 7ff40dcfb640 1 -- 172.21.15.83:0/2684327367 shutdown_connections 2024-03-25T19:23:34.077 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:34.075+0000 7ff40dcfb640 1 -- 172.21.15.83:0/2684327367 >> 172.21.15.83:0/2684327367 conn(0x7ff4080617a0 msgr2=0x7ff40813c2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:34.077 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:34.075+0000 7ff40dcfb640 1 -- 172.21.15.83:0/2684327367 shutdown_connections 2024-03-25T19:23:34.077 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:34.075+0000 7ff40dcfb640 1 -- 172.21.15.83:0/2684327367 wait complete. 2024-03-25T19:23:34.077 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:34.075+0000 7ff40dcfb640 1 librados: shutdown 2024-03-25T19:23:34.090 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:23:40.896 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:23:41.016 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.016+0000 7fd9810fb640 10 monclient: get_monmap_and_config 2024-03-25T19:23:41.016 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.016+0000 7fd9810fb640 10 monclient: build_initial_monmap 2024-03-25T19:23:41.016 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.016+0000 7fd9810fb640 10 monclient: monmap: 2024-03-25T19:23:41.016 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:23:41.016 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:23:41.016 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:23:41.017244+0000 2024-03-25T19:23:41.016 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:23:41.017244+0000 2024-03-25T19:23:41.016 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:41.016 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:23:41.016 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:23:41.016 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:23:41.016 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:23:41.016 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:41.018 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.017+0000 7fd9810fb640 1 Processor -- start 2024-03-25T19:23:41.018 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.017+0000 7fd9810fb640 1 -- start start 2024-03-25T19:23:41.018 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.017+0000 7fd9810fb640 10 monclient: init 2024-03-25T19:23:41.018 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.017+0000 7fd9810fb640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:41.018 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.017+0000 7fd9810fb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:23:41.018 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.017+0000 7fd9810fb640 10 monclient(hunting): picked mon.noname-b con 0x7fd97c0b4d60 addr v1:172.21.15.83:6790/0 2024-03-25T19:23:41.018 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.017+0000 7fd9810fb640 10 monclient(hunting): picked mon.noname-a con 0x7fd97c0b1730 addr v1:172.21.15.83:6789/0 2024-03-25T19:23:41.018 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.017+0000 7fd9810fb640 10 monclient(hunting): picked mon.noname-c con 0x7fd97c14be90 addr v1:172.21.15.103:6789/0 2024-03-25T19:23:41.018 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.017+0000 7fd9810fb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd97c0af940 con 0x7fd97c0b1730 2024-03-25T19:23:41.018 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.017+0000 7fd9810fb640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd97c0b0a70 con 0x7fd97c14be90 2024-03-25T19:23:41.019 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.017+0000 7fd9810fb640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd97c0b0d50 con 0x7fd97c0b4d60 2024-03-25T19:23:41.019 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.017+0000 7fd9810fb640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:41.019 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.017+0000 7fd9810fb640 10 monclient(hunting): authenticate will time out at 6278.289537s 2024-03-25T19:23:41.032 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.032+0000 7fd97ad9d640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fd97c0b4d60 0x7fd97c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:44188/0 (socket says 172.21.15.83:44188) 2024-03-25T19:23:41.032 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.032+0000 7fd97ad9d640 1 -- 172.21.15.83:0/3118434971 learned_addr learned my addr 172.21.15.83:0/3118434971 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:23:41.043 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.044+0000 7fd979d9b640 1 -- 172.21.15.83:0/3118434971 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2765259171 0 0) 0x7fd97c0b0d50 con 0x7fd97c0b4d60 2024-03-25T19:23:41.044 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.044+0000 7fd979d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:23:41.044 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.044+0000 7fd979d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:41.044 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.044+0000 7fd979d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:41.044 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.044+0000 7fd979d9b640 10 monclient(hunting): my global_id is 4409 2024-03-25T19:23:41.044 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.044+0000 7fd979d9b640 1 -- 172.21.15.83:0/3118434971 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd964003520 con 0x7fd97c0b4d60 2024-03-25T19:23:41.049 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.049+0000 7fd979d9b640 1 -- 172.21.15.83:0/3118434971 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2263927618 0 0) 0x7fd97c0b0a70 con 0x7fd97c14be90 2024-03-25T19:23:41.049 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.049+0000 7fd979d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:23:41.049 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.049+0000 7fd979d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:41.049 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.049+0000 7fd979d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:41.049 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.049+0000 7fd979d9b640 10 monclient(hunting): my global_id is 4402 2024-03-25T19:23:41.049 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.049+0000 7fd979d9b640 1 -- 172.21.15.83:0/3118434971 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd97c0b0d50 con 0x7fd97c14be90 2024-03-25T19:23:41.067 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.067+0000 7fd979d9b640 1 -- 172.21.15.83:0/3118434971 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3844659322 0 0) 0x7fd97c0b0d50 con 0x7fd97c14be90 2024-03-25T19:23:41.067 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.067+0000 7fd979d9b640 1 -- 172.21.15.83:0/3118434971 >> v1:172.21.15.83:6790/0 conn(0x7fd97c0b4d60 legacy=0x7fd97c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:41.067 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.067+0000 7fd979d9b640 1 -- 172.21.15.83:0/3118434971 >> v1:172.21.15.83:6789/0 conn(0x7fd97c0b1730 legacy=0x7fd97c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:41.067 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.067+0000 7fd979d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:23:41.067 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.067+0000 7fd979d9b640 1 monclient: found mon.noname-c 2024-03-25T19:23:41.067 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.067+0000 7fd979d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:41.067 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.067+0000 7fd979d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.103:6789/0 2024-03-25T19:23:41.067 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.067+0000 7fd979d9b640 1 -- 172.21.15.83:0/3118434971 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd97c0b1030 con 0x7fd97c14be90 2024-03-25T19:23:41.067 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.067+0000 7fd979d9b640 10 monclient: _finish_auth 0 2024-03-25T19:23:41.067 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.067+0000 7fd979d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:23:41.067 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.067+0000 7fd979d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:41.068 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.067+0000 7fd979d9b640 1 -- 172.21.15.83:0/3118434971 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fd974001670 con 0x7fd97c14be90 2024-03-25T19:23:41.068 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.067+0000 7fd979d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:41.068 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.067+0000 7fd979d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:23:41.068 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.067+0000 7fd979d9b640 10 monclient: dump: 2024-03-25T19:23:41.068 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:23:41.068 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:23:41.068 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:41.068 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:41.068 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:41.068 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:23:41.068 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:23:41.068 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:23:41.068 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:23:41.068 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:41.069 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.067+0000 7fd9810fb640 5 monclient: authenticate success, global_id 4402 2024-03-25T19:23:41.069 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.067+0000 7fd9810fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:23:41.086 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.086+0000 7fd979d9b640 1 -- 172.21.15.83:0/3118434971 <== mon.1 v1:172.21.15.103:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd9740017d0 con 0x7fd97c14be90 2024-03-25T19:23:41.086 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.086+0000 7fd979d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:41.086 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.086+0000 7fd979d9b640 1 -- 172.21.15.83:0/3118434971 <== mon.1 v1:172.21.15.103:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fd974001ba0 con 0x7fd97c14be90 2024-03-25T19:23:41.086 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.086+0000 7fd979d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:41.086 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.086+0000 7fd979d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:23:41.086 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.086+0000 7fd979d9b640 10 monclient: dump: 2024-03-25T19:23:41.086 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:23:41.086 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:23:41.086 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:41.086 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:41.087 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:41.087 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:23:41.087 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:23:41.087 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:23:41.087 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:23:41.087 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:41.087 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.086+0000 7fd9810fb640 10 monclient: get_monmap_and_config success 2024-03-25T19:23:41.087 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.086+0000 7fd9810fb640 4 set_mon_vals no callback set 2024-03-25T19:23:41.088 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.087+0000 7fd9810fb640 10 monclient: shutdown 2024-03-25T19:23:41.088 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.087+0000 7fd9810fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:41.088 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.087+0000 7fd9810fb640 1 -- 172.21.15.83:0/3118434971 >> v1:172.21.15.103:6789/0 conn(0x7fd97c14be90 legacy=0x7fd97c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:41.088 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.087+0000 7fd9810fb640 1 -- 172.21.15.83:0/3118434971 shutdown_connections 2024-03-25T19:23:41.088 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.087+0000 7fd9810fb640 1 -- 172.21.15.83:0/3118434971 >> 172.21.15.83:0/3118434971 conn(0x7fd97c0617a0 msgr2=0x7fd97c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:41.088 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.087+0000 7fd9810fb640 1 -- 172.21.15.83:0/3118434971 shutdown_connections 2024-03-25T19:23:41.088 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.087+0000 7fd9810fb640 1 -- 172.21.15.83:0/3118434971 wait complete. 2024-03-25T19:23:41.088 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.088+0000 7fd9810fb640 10 monclient: build_initial_monmap 2024-03-25T19:23:41.088 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.088+0000 7fd9810fb640 10 monclient: monmap: 2024-03-25T19:23:41.089 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:23:41.089 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:23:41.089 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:23:41.089 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:23:41.089 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:41.089 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:23:41.089 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:23:41.089 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:23:41.089 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:23:41.089 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:41.089 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.088+0000 7fd9810fb640 1 librados: starting msgr at 2024-03-25T19:23:41.089 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.088+0000 7fd9810fb640 1 librados: starting objecter 2024-03-25T19:23:41.090 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.089+0000 7fd9810fb640 1 Processor -- start 2024-03-25T19:23:41.091 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.089+0000 7fd9810fb640 1 -- start start 2024-03-25T19:23:41.091 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.089+0000 7fd9810fb640 1 librados: setting wanted keys 2024-03-25T19:23:41.091 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.089+0000 7fd9810fb640 1 librados: calling monclient init 2024-03-25T19:23:41.091 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.089+0000 7fd9810fb640 10 monclient: init 2024-03-25T19:23:41.091 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.090+0000 7fd9810fb640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:41.091 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.090+0000 7fd9810fb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:23:41.091 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.090+0000 7fd9810fb640 10 monclient(hunting): picked mon.noname-b con 0x7fd97c0b1730 addr v1:172.21.15.103:6789/0 2024-03-25T19:23:41.091 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.090+0000 7fd9810fb640 10 monclient(hunting): picked mon.noname-a con 0x7fd97c0b4d60 addr v1:172.21.15.83:6789/0 2024-03-25T19:23:41.091 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.090+0000 7fd9810fb640 10 monclient(hunting): picked mon.noname-c con 0x7fd97c14be90 addr v1:172.21.15.83:6790/0 2024-03-25T19:23:41.091 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.090+0000 7fd9810fb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd97c139340 con 0x7fd97c0b4d60 2024-03-25T19:23:41.091 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.090+0000 7fd9810fb640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd97c1dabf0 con 0x7fd97c0b1730 2024-03-25T19:23:41.091 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.090+0000 7fd9810fb640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd97c1db9a0 con 0x7fd97c14be90 2024-03-25T19:23:41.091 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.090+0000 7fd9810fb640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:41.092 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.090+0000 7fd9810fb640 10 monclient(hunting): authenticate will time out at 6278.361984s 2024-03-25T19:23:41.092 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.091+0000 7fd97b59e640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fd97c14be90 0x7fd97c13b4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:44252/0 (socket says 172.21.15.83:44252) 2024-03-25T19:23:41.092 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.091+0000 7fd97b59e640 1 -- 172.21.15.83:0/3736402800 learned_addr learned my addr 172.21.15.83:0/3736402800 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:23:41.092 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.092+0000 7fd9637fe640 10 client.?.objecter ms_handle_connect 0x7fd97c14be90 2024-03-25T19:23:41.092 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.092+0000 7fd9637fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:41.101 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.101+0000 7fd9637fe640 10 client.?.objecter ms_handle_connect 0x7fd97c0b4d60 2024-03-25T19:23:41.101 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.101+0000 7fd9637fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:41.104 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.105+0000 7fd9637fe640 10 client.?.objecter ms_handle_connect 0x7fd97c0b1730 2024-03-25T19:23:41.105 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.105+0000 7fd9637fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:41.160 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.160+0000 7fd9637fe640 1 -- 172.21.15.83:0/3736402800 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1018912956 0 0) 0x7fd97c139340 con 0x7fd97c0b4d60 2024-03-25T19:23:41.160 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.160+0000 7fd9637fe640 10 monclient(hunting): _negotiate 2024-03-25T19:23:41.160 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.160+0000 7fd9637fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:41.160 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.160+0000 7fd9637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:41.161 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.160+0000 7fd9637fe640 10 monclient(hunting): my global_id is 4398 2024-03-25T19:23:41.161 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.160+0000 7fd9637fe640 1 -- 172.21.15.83:0/3736402800 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd968004eb0 con 0x7fd97c0b4d60 2024-03-25T19:23:41.166 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.166+0000 7fd9637fe640 1 -- 172.21.15.83:0/3736402800 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 49343935 0 0) 0x7fd97c1dabf0 con 0x7fd97c0b1730 2024-03-25T19:23:41.166 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.166+0000 7fd9637fe640 10 monclient(hunting): _negotiate 2024-03-25T19:23:41.166 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.166+0000 7fd9637fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:41.166 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.166+0000 7fd9637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:41.166 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.166+0000 7fd9637fe640 10 monclient(hunting): my global_id is 4405 2024-03-25T19:23:41.167 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.166+0000 7fd9637fe640 1 -- 172.21.15.83:0/3736402800 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd97c139340 con 0x7fd97c0b1730 2024-03-25T19:23:41.178 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.178+0000 7fd9637fe640 1 -- 172.21.15.83:0/3736402800 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3266700456 0 0) 0x7fd968004eb0 con 0x7fd97c0b4d60 2024-03-25T19:23:41.178 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.178+0000 7fd9637fe640 1 -- 172.21.15.83:0/3736402800 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd97c1dabf0 con 0x7fd97c0b4d60 2024-03-25T19:23:41.178 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.178+0000 7fd9637fe640 1 -- 172.21.15.83:0/3736402800 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fd970001930 con 0x7fd97c0b4d60 2024-03-25T19:23:41.178 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.178+0000 7fd9637fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:41.178 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.178+0000 7fd9637fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:23:41.178 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.178+0000 7fd9637fe640 10 monclient(hunting): dump: 2024-03-25T19:23:41.178 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:23:41.178 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:23:41.179 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:41.179 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:41.179 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:41.179 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:23:41.179 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:23:41.179 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:23:41.179 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:23:41.179 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:41.179 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.178+0000 7fd9637fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:23:41.187 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.187+0000 7fd9637fe640 1 -- 172.21.15.83:0/3736402800 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4129341057 0 0) 0x7fd97c139340 con 0x7fd97c0b1730 2024-03-25T19:23:41.187 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.187+0000 7fd9637fe640 1 -- 172.21.15.83:0/3736402800 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd968004eb0 con 0x7fd97c0b1730 2024-03-25T19:23:41.187 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.187+0000 7fd9637fe640 1 -- 172.21.15.83:0/3736402800 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fd96c001cd0 con 0x7fd97c0b1730 2024-03-25T19:23:41.187 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.187+0000 7fd9637fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:41.187 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.187+0000 7fd9637fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:23:41.187 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.187+0000 7fd9637fe640 10 monclient(hunting): dump: 2024-03-25T19:23:41.187 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:23:41.187 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:23:41.187 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:41.187 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:41.187 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:41.187 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:23:41.187 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:23:41.187 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:23:41.187 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:23:41.187 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:41.197 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.197+0000 7fd9637fe640 1 -- 172.21.15.83:0/3736402800 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1439345632 0 0) 0x7fd97c1dabf0 con 0x7fd97c0b4d60 2024-03-25T19:23:41.197 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.197+0000 7fd9637fe640 1 -- 172.21.15.83:0/3736402800 >> v1:172.21.15.83:6790/0 conn(0x7fd97c14be90 legacy=0x7fd97c13b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:41.197 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.197+0000 7fd9637fe640 1 -- 172.21.15.83:0/3736402800 >> v1:172.21.15.103:6789/0 conn(0x7fd97c0b1730 legacy=0x7fd97c139540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:41.197 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.197+0000 7fd9637fe640 10 monclient: _finish_hunting 0 2024-03-25T19:23:41.197 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.197+0000 7fd9637fe640 1 monclient: found mon.a 2024-03-25T19:23:41.197 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.197+0000 7fd9637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:41.197 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.197+0000 7fd9637fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:23:41.198 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.197+0000 7fd9637fe640 1 -- 172.21.15.83:0/3736402800 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd97c1dbb30 con 0x7fd97c0b4d60 2024-03-25T19:23:41.198 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.197+0000 7fd9637fe640 10 monclient: _finish_auth 0 2024-03-25T19:23:41.198 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.197+0000 7fd9637fe640 10 monclient: _check_auth_tickets 2024-03-25T19:23:41.198 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.197+0000 7fd9637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:41.198 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.197+0000 7fd9810fb640 5 monclient: authenticate success, global_id 4398 2024-03-25T19:23:41.198 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.197+0000 7fd9810fb640 10 monclient: _renew_subs 2024-03-25T19:23:41.198 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.197+0000 7fd9810fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:23:41.198 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.197+0000 7fd9810fb640 1 -- 172.21.15.83:0/3736402800 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd97c138e60 con 0x7fd97c0b4d60 2024-03-25T19:23:41.198 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.197+0000 7fd9810fb640 10 client.4398.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:23:41.198 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.197+0000 7fd9810fb640 10 monclient: _renew_subs 2024-03-25T19:23:41.198 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.197+0000 7fd9810fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:23:41.198 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.197+0000 7fd9810fb640 1 -- 172.21.15.83:0/3736402800 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd97c13bdc0 con 0x7fd97c0b4d60 2024-03-25T19:23:41.198 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.197+0000 7fd9810fb640 1 librados: init done 2024-03-25T19:23:41.199 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.198+0000 7fd9810fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:41.199 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.198+0000 7fd9810fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:41.199 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.198+0000 7fd9810fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:23:41.199 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.198+0000 7fd9810fb640 1 -- 172.21.15.83:0/3736402800 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd97c0627b0 con 0x7fd97c0b4d60 2024-03-25T19:23:41.522 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.522+0000 7fd9637fe640 1 -- 172.21.15.83:0/3736402800 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd9700035f0 con 0x7fd97c0b4d60 2024-03-25T19:23:41.522 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.522+0000 7fd9637fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:41.522 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.522+0000 7fd9637fe640 1 -- 172.21.15.83:0/3736402800 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fd970003980 con 0x7fd97c0b4d60 2024-03-25T19:23:41.522 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.522+0000 7fd9637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:41.522 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.522+0000 7fd9637fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:23:41.523 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.522+0000 7fd9637fe640 10 monclient: dump: 2024-03-25T19:23:41.523 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:23:41.523 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:23:41.523 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:41.523 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:41.523 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:41.523 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:23:41.523 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:23:41.523 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:23:41.523 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:23:41.523 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:41.523 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.522+0000 7fd97959a640 4 set_mon_vals no callback set 2024-03-25T19:23:41.524 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.523+0000 7fd9637fe640 1 -- 172.21.15.83:0/3736402800 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7fd970003ba0 con 0x7fd97c0b4d60 2024-03-25T19:23:41.524 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.524+0000 7fd97ad9d640 10 monclient: get_auth_request con 0x7fd96806df20 auth_method 0 2024-03-25T19:23:41.524 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.524+0000 7fd9637fe640 10 client.4398.objecter ms_handle_connect 0x7fd96806df20 2024-03-25T19:23:41.529 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.529+0000 7fd9637fe640 1 -- 172.21.15.83:0/3736402800 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7fd970053650 con 0x7fd97c0b4d60 2024-03-25T19:23:41.529 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.529+0000 7fd9637fe640 10 client.4398.objecter ms_dispatch 0x7fd97c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:23:41.530 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.529+0000 7fd9637fe640 3 client.4398.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:23:41.530 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.529+0000 7fd9637fe640 3 client.4398.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:23:41.530 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.529+0000 7fd9637fe640 20 client.4398.objecter dump_active .. 0 homeless 2024-03-25T19:23:41.530 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.529+0000 7fd9637fe640 1 -- 172.21.15.83:0/3736402800 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd97004f160 con 0x7fd97c0b4d60 2024-03-25T19:23:41.530 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.529+0000 7fd9637fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:41.530 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.529+0000 7fd9637fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:23:41.781 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.781+0000 7fd9810fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:41.781 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.781+0000 7fd9810fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:41.781 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.781+0000 7fd9810fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:23:41.781 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.781+0000 7fd9810fb640 1 -- 172.21.15.83:0/3736402800 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd97c13b0f0 con 0x7fd97c0b4d60 2024-03-25T19:23:41.829 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.830+0000 7fd9637fe640 1 -- 172.21.15.83:0/3736402800 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7fd970051090 con 0x7fd97c0b4d60 2024-03-25T19:23:41.830 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.830+0000 7fd9637fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:41.830 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.830+0000 7fd9637fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:23:41.830 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:23:41.830 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:23:41.833 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.832+0000 7fd9810fb640 10 librados: watch_flush enter 2024-03-25T19:23:41.833 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.832+0000 7fd9810fb640 10 librados: watch_flush exit 2024-03-25T19:23:41.833 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.832+0000 7fd9810fb640 20 client.4398.objecter shutdown clearing up homeless session... 2024-03-25T19:23:41.833 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.832+0000 7fd9810fb640 10 client.4398.objecter successfully canceled tick 2024-03-25T19:23:41.833 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.832+0000 7fd9810fb640 1 -- 172.21.15.83:0/3736402800 >> v1:172.21.15.103:6800/345072327 conn(0x7fd96806df20 legacy=0x7fd9680703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:41.833 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.832+0000 7fd9810fb640 10 monclient: shutdown 2024-03-25T19:23:41.833 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.832+0000 7fd9810fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:41.833 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.832+0000 7fd9810fb640 1 -- 172.21.15.83:0/3736402800 >> v1:172.21.15.83:6789/0 conn(0x7fd97c0b4d60 legacy=0x7fd97c1db290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:41.833 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.832+0000 7fd9810fb640 1 -- 172.21.15.83:0/3736402800 shutdown_connections 2024-03-25T19:23:41.833 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.832+0000 7fd9810fb640 1 -- 172.21.15.83:0/3736402800 >> 172.21.15.83:0/3736402800 conn(0x7fd97c0617a0 msgr2=0x7fd97c0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:41.833 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.832+0000 7fd9810fb640 1 -- 172.21.15.83:0/3736402800 shutdown_connections 2024-03-25T19:23:41.833 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.832+0000 7fd9810fb640 1 -- 172.21.15.83:0/3736402800 wait complete. 2024-03-25T19:23:41.833 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:41.833+0000 7fd9810fb640 1 librados: shutdown 2024-03-25T19:23:41.845 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:23:48.650 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:23:48.766 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.766+0000 7f9e74359640 10 monclient: get_monmap_and_config 2024-03-25T19:23:48.766 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.766+0000 7f9e74359640 10 monclient: build_initial_monmap 2024-03-25T19:23:48.766 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.766+0000 7f9e74359640 10 monclient: monmap: 2024-03-25T19:23:48.766 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:23:48.766 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:23:48.766 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:23:48.767544+0000 2024-03-25T19:23:48.766 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:23:48.767544+0000 2024-03-25T19:23:48.766 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:48.766 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:23:48.766 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:23:48.767 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:23:48.767 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:23:48.767 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:48.768 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.767+0000 7f9e74359640 1 Processor -- start 2024-03-25T19:23:48.768 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.767+0000 7f9e74359640 1 -- start start 2024-03-25T19:23:48.768 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.767+0000 7f9e74359640 10 monclient: init 2024-03-25T19:23:48.768 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.768+0000 7f9e74359640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:48.768 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.768+0000 7f9e74359640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:23:48.768 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.768+0000 7f9e74359640 10 monclient(hunting): picked mon.noname-b con 0x7f9e6c0b0070 addr v1:172.21.15.83:6790/0 2024-03-25T19:23:48.768 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.768+0000 7f9e74359640 10 monclient(hunting): picked mon.noname-a con 0x7f9e6c13c170 addr v1:172.21.15.83:6789/0 2024-03-25T19:23:48.768 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.768+0000 7f9e74359640 10 monclient(hunting): picked mon.noname-c con 0x7f9e6c06fa90 addr v1:172.21.15.103:6789/0 2024-03-25T19:23:48.768 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.768+0000 7f9e74359640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e6c0b0450 con 0x7f9e6c13c170 2024-03-25T19:23:48.769 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.768+0000 7f9e74359640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e6c071440 con 0x7f9e6c06fa90 2024-03-25T19:23:48.769 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.768+0000 7f9e74359640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e6c0b0bf0 con 0x7f9e6c0b0070 2024-03-25T19:23:48.769 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.768+0000 7f9e74359640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:48.769 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.768+0000 7f9e74359640 10 monclient(hunting): authenticate will time out at 6286.040325s 2024-03-25T19:23:48.779 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.779+0000 7f9e718f4640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f9e6c13c170 0x7f9e6c141d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:35438/0 (socket says 172.21.15.83:35438) 2024-03-25T19:23:48.779 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.779+0000 7f9e718f4640 1 -- 172.21.15.83:0/417124077 learned_addr learned my addr 172.21.15.83:0/417124077 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:23:48.791 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.791+0000 7f9e710f3640 1 -- 172.21.15.83:0/417124077 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2359346915 0 0) 0x7f9e6c0b0450 con 0x7f9e6c13c170 2024-03-25T19:23:48.791 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.791+0000 7f9e710f3640 10 monclient(hunting): _negotiate 2024-03-25T19:23:48.791 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.791+0000 7f9e710f3640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:48.791 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.791+0000 7f9e710f3640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:48.791 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.791+0000 7f9e710f3640 10 monclient(hunting): my global_id is 4401 2024-03-25T19:23:48.791 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.791+0000 7f9e710f3640 1 -- 172.21.15.83:0/417124077 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e58004630 con 0x7f9e6c13c170 2024-03-25T19:23:48.797 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.797+0000 7f9e710f3640 1 -- 172.21.15.83:0/417124077 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2816205380 0 0) 0x7f9e6c071440 con 0x7f9e6c06fa90 2024-03-25T19:23:48.797 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.797+0000 7f9e710f3640 10 monclient(hunting): _negotiate 2024-03-25T19:23:48.797 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.797+0000 7f9e710f3640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:48.797 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.797+0000 7f9e710f3640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:48.797 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.797+0000 7f9e710f3640 10 monclient(hunting): my global_id is 4408 2024-03-25T19:23:48.797 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.797+0000 7f9e710f3640 1 -- 172.21.15.83:0/417124077 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e6c0b0450 con 0x7f9e6c06fa90 2024-03-25T19:23:48.799 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.799+0000 7f9e710f3640 1 -- 172.21.15.83:0/417124077 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 671278945 0 0) 0x7f9e6c0b0bf0 con 0x7f9e6c0b0070 2024-03-25T19:23:48.799 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.799+0000 7f9e710f3640 10 monclient(hunting): _negotiate 2024-03-25T19:23:48.799 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.799+0000 7f9e710f3640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:48.799 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.799+0000 7f9e710f3640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:48.799 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.799+0000 7f9e710f3640 10 monclient(hunting): my global_id is 4415 2024-03-25T19:23:48.799 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.799+0000 7f9e710f3640 1 -- 172.21.15.83:0/417124077 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e6c071440 con 0x7f9e6c0b0070 2024-03-25T19:23:48.801 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.801+0000 7f9e710f3640 1 -- 172.21.15.83:0/417124077 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 616925049 0 0) 0x7f9e58004630 con 0x7f9e6c13c170 2024-03-25T19:23:48.801 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.801+0000 7f9e710f3640 1 -- 172.21.15.83:0/417124077 >> v1:172.21.15.83:6790/0 conn(0x7f9e6c0b0070 legacy=0x7f9e6c13ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:48.801 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.801+0000 7f9e710f3640 1 -- 172.21.15.83:0/417124077 >> v1:172.21.15.103:6789/0 conn(0x7f9e6c06fa90 legacy=0x7f9e6c14a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:48.801 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.801+0000 7f9e710f3640 10 monclient: _finish_hunting 0 2024-03-25T19:23:48.801 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.801+0000 7f9e710f3640 1 monclient: found mon.noname-a 2024-03-25T19:23:48.801 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.801+0000 7f9e710f3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:48.801 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.801+0000 7f9e710f3640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.83:6789/0 2024-03-25T19:23:48.801 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.801+0000 7f9e710f3640 1 -- 172.21.15.83:0/417124077 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e6c0716b0 con 0x7f9e6c13c170 2024-03-25T19:23:48.801 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.801+0000 7f9e710f3640 10 monclient: _finish_auth 0 2024-03-25T19:23:48.801 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.801+0000 7f9e710f3640 10 monclient: _check_auth_tickets 2024-03-25T19:23:48.801 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.801+0000 7f9e710f3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:48.802 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.801+0000 7f9e710f3640 1 -- 172.21.15.83:0/417124077 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f9e5c001690 con 0x7f9e6c13c170 2024-03-25T19:23:48.802 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.801+0000 7f9e710f3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:48.802 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.801+0000 7f9e710f3640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:23:48.802 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.801+0000 7f9e710f3640 10 monclient: dump: 2024-03-25T19:23:48.802 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:23:48.802 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:23:48.802 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:48.802 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:48.802 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:48.802 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:23:48.802 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:23:48.802 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:23:48.802 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:23:48.802 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:48.803 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.801+0000 7f9e74359640 5 monclient: authenticate success, global_id 4401 2024-03-25T19:23:48.803 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.801+0000 7f9e74359640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:23:48.816 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.816+0000 7f9e710f3640 1 -- 172.21.15.83:0/417124077 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9e5c0017f0 con 0x7f9e6c13c170 2024-03-25T19:23:48.816 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.816+0000 7f9e710f3640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:48.816 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.816+0000 7f9e710f3640 1 -- 172.21.15.83:0/417124077 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f9e5c001c00 con 0x7f9e6c13c170 2024-03-25T19:23:48.816 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.816+0000 7f9e74359640 10 monclient: get_monmap_and_config success 2024-03-25T19:23:48.816 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.816+0000 7f9e74359640 4 set_mon_vals no callback set 2024-03-25T19:23:48.816 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.816+0000 7f9e710f3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:48.816 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.816+0000 7f9e710f3640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:23:48.816 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.816+0000 7f9e710f3640 10 monclient: dump: 2024-03-25T19:23:48.816 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:23:48.816 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:23:48.816 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:48.816 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:48.816 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:48.816 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:23:48.817 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:23:48.817 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:23:48.817 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:23:48.817 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:48.819 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.817+0000 7f9e74359640 10 monclient: shutdown 2024-03-25T19:23:48.819 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.817+0000 7f9e74359640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:48.819 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.817+0000 7f9e74359640 1 -- 172.21.15.83:0/417124077 >> v1:172.21.15.83:6789/0 conn(0x7f9e6c13c170 legacy=0x7f9e6c141d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:48.819 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.817+0000 7f9e74359640 1 -- 172.21.15.83:0/417124077 shutdown_connections 2024-03-25T19:23:48.819 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.817+0000 7f9e74359640 1 -- 172.21.15.83:0/417124077 >> 172.21.15.83:0/417124077 conn(0x7f9e6c0b0ef0 msgr2=0x7f9e6c0b12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:48.819 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.817+0000 7f9e74359640 1 -- 172.21.15.83:0/417124077 shutdown_connections 2024-03-25T19:23:48.819 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.817+0000 7f9e74359640 1 -- 172.21.15.83:0/417124077 wait complete. 2024-03-25T19:23:48.819 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.818+0000 7f9e74359640 10 monclient: build_initial_monmap 2024-03-25T19:23:48.819 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.818+0000 7f9e74359640 10 monclient: monmap: 2024-03-25T19:23:48.819 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:23:48.819 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:23:48.819 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:23:48.820 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:23:48.820 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:48.820 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:23:48.820 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:23:48.820 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:23:48.820 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:23:48.820 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:48.820 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.818+0000 7f9e74359640 1 librados: starting msgr at 2024-03-25T19:23:48.820 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.818+0000 7f9e74359640 1 librados: starting objecter 2024-03-25T19:23:48.820 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.818+0000 7f9e74359640 1 Processor -- start 2024-03-25T19:23:48.820 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.818+0000 7f9e74359640 1 -- start start 2024-03-25T19:23:48.820 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.818+0000 7f9e74359640 1 librados: setting wanted keys 2024-03-25T19:23:48.820 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.818+0000 7f9e74359640 1 librados: calling monclient init 2024-03-25T19:23:48.820 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.818+0000 7f9e74359640 10 monclient: init 2024-03-25T19:23:48.820 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.819+0000 7f9e74359640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:48.821 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.819+0000 7f9e74359640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:23:48.821 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.819+0000 7f9e74359640 10 monclient(hunting): picked mon.noname-b con 0x7f9e6c06fa90 addr v1:172.21.15.103:6789/0 2024-03-25T19:23:48.821 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.819+0000 7f9e74359640 10 monclient(hunting): picked mon.noname-a con 0x7f9e6c0b0070 addr v1:172.21.15.83:6789/0 2024-03-25T19:23:48.821 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.819+0000 7f9e74359640 10 monclient(hunting): picked mon.noname-c con 0x7f9e6c13c170 addr v1:172.21.15.83:6790/0 2024-03-25T19:23:48.821 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.819+0000 7f9e74359640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e6c0a96c0 con 0x7f9e6c0b0070 2024-03-25T19:23:48.821 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.819+0000 7f9e74359640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e6c1c9d10 con 0x7f9e6c06fa90 2024-03-25T19:23:48.821 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.819+0000 7f9e74359640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e6c1c9ea0 con 0x7f9e6c13c170 2024-03-25T19:23:48.821 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.819+0000 7f9e74359640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:48.821 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.819+0000 7f9e74359640 10 monclient(hunting): authenticate will time out at 6286.091124s 2024-03-25T19:23:48.836 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.837+0000 7f9e728f6640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f9e6c13c170 0x7f9e6c13de20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:58820/0 (socket says 172.21.15.83:58820) 2024-03-25T19:23:48.836 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.837+0000 7f9e728f6640 1 -- 172.21.15.83:0/3857831872 learned_addr learned my addr 172.21.15.83:0/3857831872 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:23:48.837 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.838+0000 7f9e62ffd640 10 client.?.objecter ms_handle_connect 0x7f9e6c13c170 2024-03-25T19:23:48.838 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.838+0000 7f9e62ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:48.838 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.839+0000 7f9e62ffd640 10 client.?.objecter ms_handle_connect 0x7f9e6c0b0070 2024-03-25T19:23:48.839 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.839+0000 7f9e62ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:48.841 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.841+0000 7f9e62ffd640 10 client.?.objecter ms_handle_connect 0x7f9e6c06fa90 2024-03-25T19:23:48.841 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.841+0000 7f9e62ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:48.846 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.847+0000 7f9e62ffd640 1 -- 172.21.15.83:0/3857831872 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3745448039 0 0) 0x7f9e6c1c9ea0 con 0x7f9e6c13c170 2024-03-25T19:23:48.847 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.847+0000 7f9e62ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:23:48.847 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.847+0000 7f9e62ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:48.847 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.847+0000 7f9e62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:48.847 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.847+0000 7f9e62ffd640 10 monclient(hunting): my global_id is 4418 2024-03-25T19:23:48.847 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.847+0000 7f9e62ffd640 1 -- 172.21.15.83:0/3857831872 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e50004eb0 con 0x7f9e6c13c170 2024-03-25T19:23:48.850 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.850+0000 7f9e62ffd640 1 -- 172.21.15.83:0/3857831872 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 851305219 0 0) 0x7f9e6c1c9d10 con 0x7f9e6c06fa90 2024-03-25T19:23:48.850 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.850+0000 7f9e62ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:23:48.850 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.850+0000 7f9e62ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:48.850 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.850+0000 7f9e62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:48.850 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.850+0000 7f9e62ffd640 10 monclient(hunting): my global_id is 4411 2024-03-25T19:23:48.850 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.851+0000 7f9e62ffd640 1 -- 172.21.15.83:0/3857831872 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e6c1c9ea0 con 0x7f9e6c06fa90 2024-03-25T19:23:48.856 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.857+0000 7f9e62ffd640 1 -- 172.21.15.83:0/3857831872 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3918446302 0 0) 0x7f9e6c0a96c0 con 0x7f9e6c0b0070 2024-03-25T19:23:48.856 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.857+0000 7f9e62ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:23:48.856 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.857+0000 7f9e62ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:48.856 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.857+0000 7f9e62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:48.857 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.857+0000 7f9e62ffd640 10 monclient(hunting): my global_id is 4404 2024-03-25T19:23:48.857 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.857+0000 7f9e62ffd640 1 -- 172.21.15.83:0/3857831872 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e6c1c9d10 con 0x7f9e6c0b0070 2024-03-25T19:23:48.870 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.870+0000 7f9e62ffd640 1 -- 172.21.15.83:0/3857831872 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1920939021 0 0) 0x7f9e6c1c9d10 con 0x7f9e6c0b0070 2024-03-25T19:23:48.871 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.870+0000 7f9e62ffd640 1 -- 172.21.15.83:0/3857831872 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e6c0a96c0 con 0x7f9e6c0b0070 2024-03-25T19:23:48.871 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.870+0000 7f9e62ffd640 1 -- 172.21.15.83:0/3857831872 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f9e5c001b00 con 0x7f9e6c0b0070 2024-03-25T19:23:48.871 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.870+0000 7f9e62ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:48.871 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.870+0000 7f9e62ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:23:48.871 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.870+0000 7f9e62ffd640 10 monclient(hunting): dump: 2024-03-25T19:23:48.871 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:23:48.871 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:23:48.871 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:48.871 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:48.871 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:48.871 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:23:48.871 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:23:48.871 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:23:48.871 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:23:48.872 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:48.872 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.870+0000 7f9e62ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:23:48.872 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.871+0000 7f9e62ffd640 1 -- 172.21.15.83:0/3857831872 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3425358724 0 0) 0x7f9e6c1c9ea0 con 0x7f9e6c06fa90 2024-03-25T19:23:48.872 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.871+0000 7f9e62ffd640 1 -- 172.21.15.83:0/3857831872 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e6c1c9d10 con 0x7f9e6c06fa90 2024-03-25T19:23:48.872 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.871+0000 7f9e62ffd640 1 -- 172.21.15.83:0/3857831872 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f9e68001870 con 0x7f9e6c06fa90 2024-03-25T19:23:48.872 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.871+0000 7f9e62ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:48.872 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.871+0000 7f9e62ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:23:48.872 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.871+0000 7f9e62ffd640 10 monclient(hunting): dump: 2024-03-25T19:23:48.872 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:23:48.872 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:23:48.872 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:48.872 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:48.872 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:48.872 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:23:48.872 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:23:48.873 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:23:48.873 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:23:48.873 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:48.881 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.880+0000 7f9e62ffd640 1 -- 172.21.15.83:0/3857831872 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3816724996 0 0) 0x7f9e50004eb0 con 0x7f9e6c13c170 2024-03-25T19:23:48.881 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.880+0000 7f9e62ffd640 1 -- 172.21.15.83:0/3857831872 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e6c1c9ea0 con 0x7f9e6c13c170 2024-03-25T19:23:48.881 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.880+0000 7f9e62ffd640 1 -- 172.21.15.83:0/3857831872 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f9e64001d30 con 0x7f9e6c13c170 2024-03-25T19:23:48.881 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.880+0000 7f9e62ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:48.881 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.880+0000 7f9e62ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:23:48.881 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.880+0000 7f9e62ffd640 10 monclient(hunting): dump: 2024-03-25T19:23:48.881 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:23:48.881 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:23:48.881 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:48.881 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:48.882 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:48.882 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:23:48.882 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:23:48.882 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:23:48.882 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:23:48.882 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:48.882 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.880+0000 7f9e62ffd640 1 -- 172.21.15.83:0/3857831872 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2710584745 0 0) 0x7f9e6c0a96c0 con 0x7f9e6c0b0070 2024-03-25T19:23:48.882 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.880+0000 7f9e62ffd640 1 -- 172.21.15.83:0/3857831872 >> v1:172.21.15.83:6790/0 conn(0x7f9e6c13c170 legacy=0x7f9e6c13de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:48.882 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.880+0000 7f9e62ffd640 1 -- 172.21.15.83:0/3857831872 >> v1:172.21.15.103:6789/0 conn(0x7f9e6c06fa90 legacy=0x7f9e6c1d8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:48.882 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.880+0000 7f9e62ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:23:48.882 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.880+0000 7f9e62ffd640 1 monclient: found mon.a 2024-03-25T19:23:48.882 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.880+0000 7f9e62ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:48.882 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.880+0000 7f9e62ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:23:48.882 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.880+0000 7f9e62ffd640 1 -- 172.21.15.83:0/3857831872 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e6c1d9610 con 0x7f9e6c0b0070 2024-03-25T19:23:48.883 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.880+0000 7f9e62ffd640 10 monclient: _finish_auth 0 2024-03-25T19:23:48.883 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.880+0000 7f9e62ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:23:48.883 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.880+0000 7f9e62ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:48.883 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.880+0000 7f9e74359640 5 monclient: authenticate success, global_id 4404 2024-03-25T19:23:48.883 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.880+0000 7f9e74359640 10 monclient: _renew_subs 2024-03-25T19:23:48.883 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.880+0000 7f9e74359640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:23:48.883 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.880+0000 7f9e74359640 1 -- 172.21.15.83:0/3857831872 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e6c1d9880 con 0x7f9e6c0b0070 2024-03-25T19:23:48.883 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.880+0000 7f9e74359640 10 client.4404.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:23:48.883 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.880+0000 7f9e74359640 10 monclient: _renew_subs 2024-03-25T19:23:48.883 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.880+0000 7f9e74359640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:23:48.883 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.880+0000 7f9e74359640 1 -- 172.21.15.83:0/3857831872 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e6c136f20 con 0x7f9e6c0b0070 2024-03-25T19:23:48.883 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.880+0000 7f9e74359640 1 librados: init done 2024-03-25T19:23:48.883 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.881+0000 7f9e74359640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:48.883 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.881+0000 7f9e74359640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:48.883 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.881+0000 7f9e74359640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:23:48.884 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:48.881+0000 7f9e74359640 1 -- 172.21.15.83:0/3857831872 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e6c0627b0 con 0x7f9e6c0b0070 2024-03-25T19:23:49.185 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:49.185+0000 7f9e62ffd640 1 -- 172.21.15.83:0/3857831872 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9e5c001240 con 0x7f9e6c0b0070 2024-03-25T19:23:49.185 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:49.185+0000 7f9e62ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:49.185 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:49.185+0000 7f9e62ffd640 1 -- 172.21.15.83:0/3857831872 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f9e5c001c60 con 0x7f9e6c0b0070 2024-03-25T19:23:49.185 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:49.185+0000 7f9e62ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:49.185 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:49.185+0000 7f9e62ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:23:49.185 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:49.185+0000 7f9e62ffd640 10 monclient: dump: 2024-03-25T19:23:49.185 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:23:49.185 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:23:49.185 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:49.185 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:49.186 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:49.186 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:23:49.186 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:23:49.186 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:23:49.186 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:23:49.186 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:49.186 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:49.185+0000 7f9e708f2640 4 set_mon_vals no callback set 2024-03-25T19:23:49.187 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:49.186+0000 7f9e62ffd640 1 -- 172.21.15.83:0/3857831872 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7f9e5c01a900 con 0x7f9e6c0b0070 2024-03-25T19:23:49.187 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:49.186+0000 7f9e720f5640 10 monclient: get_auth_request con 0x7f9e5006e0f0 auth_method 0 2024-03-25T19:23:49.187 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:49.186+0000 7f9e62ffd640 10 client.4404.objecter ms_handle_connect 0x7f9e5006e0f0 2024-03-25T19:23:49.195 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:49.195+0000 7f9e62ffd640 1 -- 172.21.15.83:0/3857831872 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7f9e5c0544f0 con 0x7f9e6c0b0070 2024-03-25T19:23:49.195 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:49.195+0000 7f9e62ffd640 10 client.4404.objecter ms_dispatch 0x7f9e6c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:23:49.195 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:49.195+0000 7f9e62ffd640 3 client.4404.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:23:49.195 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:49.195+0000 7f9e62ffd640 3 client.4404.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:23:49.195 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:49.195+0000 7f9e62ffd640 20 client.4404.objecter dump_active .. 0 homeless 2024-03-25T19:23:49.195 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:49.195+0000 7f9e62ffd640 1 -- 172.21.15.83:0/3857831872 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9e5c050000 con 0x7f9e6c0b0070 2024-03-25T19:23:49.195 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:49.195+0000 7f9e62ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:49.195 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:49.195+0000 7f9e62ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:23:49.417 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:49.417+0000 7f9e74359640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:49.417 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:49.417+0000 7f9e74359640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:49.417 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:49.417+0000 7f9e74359640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:23:49.417 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:49.417+0000 7f9e74359640 1 -- 172.21.15.83:0/3857831872 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9e6c0ae550 con 0x7f9e6c0b0070 2024-03-25T19:23:49.453 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:49.453+0000 7f9e62ffd640 1 -- 172.21.15.83:0/3857831872 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7f9e5c051f30 con 0x7f9e6c0b0070 2024-03-25T19:23:49.453 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:49.453+0000 7f9e62ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:49.453 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:49.453+0000 7f9e62ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:23:49.453 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:23:49.454 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:23:49.456 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:49.455+0000 7f9e74359640 10 librados: watch_flush enter 2024-03-25T19:23:49.456 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:49.455+0000 7f9e74359640 10 librados: watch_flush exit 2024-03-25T19:23:49.456 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:49.455+0000 7f9e74359640 20 client.4404.objecter shutdown clearing up homeless session... 2024-03-25T19:23:49.456 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:49.455+0000 7f9e74359640 10 client.4404.objecter successfully canceled tick 2024-03-25T19:23:49.456 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:49.456+0000 7f9e74359640 1 -- 172.21.15.83:0/3857831872 >> v1:172.21.15.103:6800/345072327 conn(0x7f9e5006e0f0 legacy=0x7f9e500705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:49.456 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:49.456+0000 7f9e74359640 10 monclient: shutdown 2024-03-25T19:23:49.456 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:49.456+0000 7f9e74359640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:49.456 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:49.456+0000 7f9e74359640 1 -- 172.21.15.83:0/3857831872 >> v1:172.21.15.83:6789/0 conn(0x7f9e6c0b0070 legacy=0x7f9e6c136630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:49.457 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:49.456+0000 7f9e74359640 1 -- 172.21.15.83:0/3857831872 shutdown_connections 2024-03-25T19:23:49.457 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:49.456+0000 7f9e74359640 1 -- 172.21.15.83:0/3857831872 >> 172.21.15.83:0/3857831872 conn(0x7f9e6c0b0ef0 msgr2=0x7f9e6c0615c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:49.457 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:49.456+0000 7f9e74359640 1 -- 172.21.15.83:0/3857831872 shutdown_connections 2024-03-25T19:23:49.457 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:49.456+0000 7f9e74359640 1 -- 172.21.15.83:0/3857831872 wait complete. 2024-03-25T19:23:49.457 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:49.456+0000 7f9e74359640 1 librados: shutdown 2024-03-25T19:23:49.467 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:23:56.274 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:23:56.389 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.389+0000 7f79c5463640 10 monclient: get_monmap_and_config 2024-03-25T19:23:56.389 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.389+0000 7f79c5463640 10 monclient: build_initial_monmap 2024-03-25T19:23:56.389 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.389+0000 7f79c5463640 10 monclient: monmap: 2024-03-25T19:23:56.389 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:23:56.389 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:23:56.389 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:23:56.390593+0000 2024-03-25T19:23:56.389 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:23:56.390593+0000 2024-03-25T19:23:56.389 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:56.389 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:23:56.389 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:23:56.390 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:23:56.390 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:23:56.390 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:56.391 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.390+0000 7f79c5463640 1 Processor -- start 2024-03-25T19:23:56.391 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.390+0000 7f79c5463640 1 -- start start 2024-03-25T19:23:56.391 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.390+0000 7f79c5463640 10 monclient: init 2024-03-25T19:23:56.391 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.391+0000 7f79c5463640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:56.391 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.391+0000 7f79c5463640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:23:56.391 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.391+0000 7f79c5463640 10 monclient(hunting): picked mon.noname-c con 0x7f79c00a98f0 addr v1:172.21.15.103:6789/0 2024-03-25T19:23:56.391 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.391+0000 7f79c5463640 10 monclient(hunting): picked mon.noname-a con 0x7f79c013f710 addr v1:172.21.15.83:6789/0 2024-03-25T19:23:56.391 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.391+0000 7f79c5463640 10 monclient(hunting): picked mon.noname-b con 0x7f79c01430c0 addr v1:172.21.15.83:6790/0 2024-03-25T19:23:56.391 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.391+0000 7f79c5463640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79c0142520 con 0x7f79c013f710 2024-03-25T19:23:56.391 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.391+0000 7f79c5463640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79c00b28a0 con 0x7f79c00a98f0 2024-03-25T19:23:56.391 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.391+0000 7f79c5463640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79c00b2a30 con 0x7f79c01430c0 2024-03-25T19:23:56.392 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.391+0000 7f79c5463640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:56.392 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.391+0000 7f79c5463640 10 monclient(hunting): authenticate will time out at 6293.663331s 2024-03-25T19:23:56.395 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.396+0000 7f79beffd640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f79c00a98f0 0x7f79c00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:58588/0 (socket says 172.21.15.83:58588) 2024-03-25T19:23:56.396 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.396+0000 7f79beffd640 1 -- 172.21.15.83:0/1895854981 learned_addr learned my addr 172.21.15.83:0/1895854981 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:23:56.417 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.417+0000 7f79bdffb640 1 -- 172.21.15.83:0/1895854981 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3665040678 0 0) 0x7f79c0142520 con 0x7f79c013f710 2024-03-25T19:23:56.417 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.417+0000 7f79bdffb640 10 monclient(hunting): _negotiate 2024-03-25T19:23:56.417 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.417+0000 7f79bdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:56.417 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.417+0000 7f79bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:56.417 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.417+0000 7f79bdffb640 10 monclient(hunting): my global_id is 4407 2024-03-25T19:23:56.417 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.417+0000 7f79bdffb640 1 -- 172.21.15.83:0/1895854981 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79a8003520 con 0x7f79c013f710 2024-03-25T19:23:56.424 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.424+0000 7f79bdffb640 1 -- 172.21.15.83:0/1895854981 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 769749750 0 0) 0x7f79c00b28a0 con 0x7f79c00a98f0 2024-03-25T19:23:56.424 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.424+0000 7f79bdffb640 10 monclient(hunting): _negotiate 2024-03-25T19:23:56.424 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.424+0000 7f79bdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:56.424 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.424+0000 7f79bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:56.424 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.424+0000 7f79bdffb640 10 monclient(hunting): my global_id is 4414 2024-03-25T19:23:56.424 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.424+0000 7f79bdffb640 1 -- 172.21.15.83:0/1895854981 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79c0142520 con 0x7f79c00a98f0 2024-03-25T19:23:56.426 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.427+0000 7f79bdffb640 1 -- 172.21.15.83:0/1895854981 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3958621233 0 0) 0x7f79c00b2a30 con 0x7f79c01430c0 2024-03-25T19:23:56.426 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.427+0000 7f79bdffb640 10 monclient(hunting): _negotiate 2024-03-25T19:23:56.426 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.427+0000 7f79bdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:56.426 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.427+0000 7f79bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:56.427 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.427+0000 7f79bdffb640 10 monclient(hunting): my global_id is 4421 2024-03-25T19:23:56.427 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.427+0000 7f79bdffb640 1 -- 172.21.15.83:0/1895854981 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79c00b28a0 con 0x7f79c01430c0 2024-03-25T19:23:56.440 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.440+0000 7f79bdffb640 1 -- 172.21.15.83:0/1895854981 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1038023616 0 0) 0x7f79c00b28a0 con 0x7f79c01430c0 2024-03-25T19:23:56.440 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.440+0000 7f79bdffb640 1 -- 172.21.15.83:0/1895854981 >> v1:172.21.15.103:6789/0 conn(0x7f79c00a98f0 legacy=0x7f79c00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:56.440 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.440+0000 7f79bdffb640 1 -- 172.21.15.83:0/1895854981 >> v1:172.21.15.83:6789/0 conn(0x7f79c013f710 legacy=0x7f79c01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:56.440 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.440+0000 7f79bdffb640 10 monclient: _finish_hunting 0 2024-03-25T19:23:56.440 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.440+0000 7f79bdffb640 1 monclient: found mon.noname-b 2024-03-25T19:23:56.440 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.440+0000 7f79bdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:56.440 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.440+0000 7f79bdffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.83:6790/0 2024-03-25T19:23:56.440 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.440+0000 7f79bdffb640 1 -- 172.21.15.83:0/1895854981 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79c00b44c0 con 0x7f79c01430c0 2024-03-25T19:23:56.441 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.440+0000 7f79bdffb640 10 monclient: _finish_auth 0 2024-03-25T19:23:56.441 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.440+0000 7f79bdffb640 10 monclient: _check_auth_tickets 2024-03-25T19:23:56.441 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.440+0000 7f79bdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:56.441 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.440+0000 7f79bdffb640 1 -- 172.21.15.83:0/1895854981 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f79b8001930 con 0x7f79c01430c0 2024-03-25T19:23:56.441 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.440+0000 7f79c5463640 5 monclient: authenticate success, global_id 4421 2024-03-25T19:23:56.441 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.440+0000 7f79c5463640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:23:56.441 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.440+0000 7f79bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:56.441 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.440+0000 7f79bdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:23:56.441 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.440+0000 7f79bdffb640 10 monclient: dump: 2024-03-25T19:23:56.441 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:23:56.441 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:23:56.441 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:56.441 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:56.441 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:56.441 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:23:56.442 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:23:56.442 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:23:56.442 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:23:56.442 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:56.442 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.440+0000 7f79c5463640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:23:56.454 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.454+0000 7f79bdffb640 1 -- 172.21.15.83:0/1895854981 <== mon.2 v1:172.21.15.83:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f79b8001a90 con 0x7f79c01430c0 2024-03-25T19:23:56.454 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.454+0000 7f79bdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:56.454 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.454+0000 7f79bdffb640 1 -- 172.21.15.83:0/1895854981 <== mon.2 v1:172.21.15.83:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f79b8001ee0 con 0x7f79c01430c0 2024-03-25T19:23:56.454 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.454+0000 7f79c5463640 10 monclient: get_monmap_and_config success 2024-03-25T19:23:56.454 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.454+0000 7f79c5463640 4 set_mon_vals no callback set 2024-03-25T19:23:56.454 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.454+0000 7f79bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:56.454 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.454+0000 7f79bdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:23:56.454 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.454+0000 7f79bdffb640 10 monclient: dump: 2024-03-25T19:23:56.454 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:23:56.454 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:23:56.455 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:56.455 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:56.455 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:56.455 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:23:56.455 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:23:56.455 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:23:56.455 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:23:56.455 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:56.457 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.455+0000 7f79c5463640 10 monclient: shutdown 2024-03-25T19:23:56.457 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.455+0000 7f79c5463640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:56.457 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.455+0000 7f79c5463640 1 -- 172.21.15.83:0/1895854981 >> v1:172.21.15.83:6790/0 conn(0x7f79c01430c0 legacy=0x7f79c014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:56.457 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.455+0000 7f79c5463640 1 -- 172.21.15.83:0/1895854981 shutdown_connections 2024-03-25T19:23:56.457 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.455+0000 7f79c5463640 1 -- 172.21.15.83:0/1895854981 >> 172.21.15.83:0/1895854981 conn(0x7f79c00b7ea0 msgr2=0x7f79c00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:56.457 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.455+0000 7f79c5463640 1 -- 172.21.15.83:0/1895854981 shutdown_connections 2024-03-25T19:23:56.457 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.455+0000 7f79c5463640 1 -- 172.21.15.83:0/1895854981 wait complete. 2024-03-25T19:23:56.457 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.456+0000 7f79c5463640 10 monclient: build_initial_monmap 2024-03-25T19:23:56.458 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.456+0000 7f79c5463640 10 monclient: monmap: 2024-03-25T19:23:56.458 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:23:56.458 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:23:56.458 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:23:56.458 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:23:56.458 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:56.458 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:23:56.458 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:23:56.458 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:23:56.458 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:23:56.458 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:56.458 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.457+0000 7f79c5463640 1 librados: starting msgr at 2024-03-25T19:23:56.458 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.457+0000 7f79c5463640 1 librados: starting objecter 2024-03-25T19:23:56.459 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.457+0000 7f79c5463640 1 Processor -- start 2024-03-25T19:23:56.459 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.457+0000 7f79c5463640 1 -- start start 2024-03-25T19:23:56.459 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.457+0000 7f79c5463640 1 librados: setting wanted keys 2024-03-25T19:23:56.459 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.457+0000 7f79c5463640 1 librados: calling monclient init 2024-03-25T19:23:56.459 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.457+0000 7f79c5463640 10 monclient: init 2024-03-25T19:23:56.460 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.458+0000 7f79c5463640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:56.460 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.458+0000 7f79c5463640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:23:56.460 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.458+0000 7f79c5463640 10 monclient(hunting): picked mon.noname-c con 0x7f79c00a98f0 addr v1:172.21.15.83:6790/0 2024-03-25T19:23:56.460 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.458+0000 7f79c5463640 10 monclient(hunting): picked mon.noname-a con 0x7f79c013f710 addr v1:172.21.15.83:6789/0 2024-03-25T19:23:56.460 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.458+0000 7f79c5463640 10 monclient(hunting): picked mon.noname-b con 0x7f79c01430c0 addr v1:172.21.15.103:6789/0 2024-03-25T19:23:56.460 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.458+0000 7f79c5463640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79c01349e0 con 0x7f79c013f710 2024-03-25T19:23:56.460 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.458+0000 7f79c5463640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79c0138600 con 0x7f79c01430c0 2024-03-25T19:23:56.460 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.458+0000 7f79c5463640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79c00b1770 con 0x7f79c00a98f0 2024-03-25T19:23:56.460 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.458+0000 7f79c5463640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:56.460 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.458+0000 7f79c5463640 10 monclient(hunting): authenticate will time out at 6293.730558s 2024-03-25T19:23:56.461 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.462+0000 7f79be7fc640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f79c013f710 0x7f79c0133bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:37910/0 (socket says 172.21.15.83:37910) 2024-03-25T19:23:56.462 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.462+0000 7f79be7fc640 1 -- 172.21.15.83:0/2497886097 learned_addr learned my addr 172.21.15.83:0/2497886097 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:23:56.464 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.465+0000 7f79a77fe640 10 client.?.objecter ms_handle_connect 0x7f79c013f710 2024-03-25T19:23:56.464 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.465+0000 7f79a77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:56.473 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.474+0000 7f79a77fe640 10 client.?.objecter ms_handle_connect 0x7f79c00a98f0 2024-03-25T19:23:56.473 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.474+0000 7f79a77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:56.476 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.476+0000 7f79a77fe640 1 -- 172.21.15.83:0/2497886097 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1040166728 0 0) 0x7f79c01349e0 con 0x7f79c013f710 2024-03-25T19:23:56.476 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.476+0000 7f79a77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:23:56.476 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.476+0000 7f79a77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:56.476 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.476+0000 7f79a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:56.476 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.476+0000 7f79a77fe640 10 monclient(hunting): my global_id is 4410 2024-03-25T19:23:56.476 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.476+0000 7f79a77fe640 1 -- 172.21.15.83:0/2497886097 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79ac004eb0 con 0x7f79c013f710 2024-03-25T19:23:56.480 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.481+0000 7f79a77fe640 10 client.?.objecter ms_handle_connect 0x7f79c01430c0 2024-03-25T19:23:56.481 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.481+0000 7f79a77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:56.481 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.482+0000 7f79a77fe640 1 -- 172.21.15.83:0/2497886097 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1869983089 0 0) 0x7f79c00b1770 con 0x7f79c00a98f0 2024-03-25T19:23:56.481 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.482+0000 7f79a77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:23:56.481 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.482+0000 7f79a77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:56.481 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.482+0000 7f79a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:56.481 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.482+0000 7f79a77fe640 10 monclient(hunting): my global_id is 4424 2024-03-25T19:23:56.482 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.482+0000 7f79a77fe640 1 -- 172.21.15.83:0/2497886097 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79c01349e0 con 0x7f79c00a98f0 2024-03-25T19:23:56.488 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.489+0000 7f79a77fe640 1 -- 172.21.15.83:0/2497886097 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3490364958 0 0) 0x7f79c01349e0 con 0x7f79c00a98f0 2024-03-25T19:23:56.488 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.489+0000 7f79a77fe640 1 -- 172.21.15.83:0/2497886097 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f79c00b1770 con 0x7f79c00a98f0 2024-03-25T19:23:56.491 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.491+0000 7f79a77fe640 1 -- 172.21.15.83:0/2497886097 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2369943458 0 0) 0x7f79c0138600 con 0x7f79c01430c0 2024-03-25T19:23:56.491 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.491+0000 7f79a77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:23:56.491 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.491+0000 7f79a77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:56.491 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.491+0000 7f79a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:56.491 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.491+0000 7f79a77fe640 10 monclient(hunting): my global_id is 4417 2024-03-25T19:23:56.492 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.491+0000 7f79a77fe640 1 -- 172.21.15.83:0/2497886097 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79c01349e0 con 0x7f79c01430c0 2024-03-25T19:23:56.492 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.491+0000 7f79a77fe640 1 -- 172.21.15.83:0/2497886097 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 643670019 0 0) 0x7f79ac004eb0 con 0x7f79c013f710 2024-03-25T19:23:56.492 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.491+0000 7f79a77fe640 1 -- 172.21.15.83:0/2497886097 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f79c0138600 con 0x7f79c013f710 2024-03-25T19:23:56.492 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.491+0000 7f79a77fe640 1 -- 172.21.15.83:0/2497886097 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f79b40019b0 con 0x7f79c013f710 2024-03-25T19:23:56.492 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.491+0000 7f79a77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:56.492 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.491+0000 7f79a77fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:23:56.492 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.491+0000 7f79a77fe640 10 monclient(hunting): dump: 2024-03-25T19:23:56.492 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:23:56.492 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:23:56.492 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:56.492 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:56.492 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:56.492 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:23:56.492 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:23:56.493 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:23:56.493 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:23:56.493 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:56.493 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.491+0000 7f79a77fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:23:56.502 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.503+0000 7f79a77fe640 1 -- 172.21.15.83:0/2497886097 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2299827511 0 0) 0x7f79c01349e0 con 0x7f79c01430c0 2024-03-25T19:23:56.502 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.503+0000 7f79a77fe640 1 -- 172.21.15.83:0/2497886097 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f79ac004eb0 con 0x7f79c01430c0 2024-03-25T19:23:56.502 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.503+0000 7f79a77fe640 1 -- 172.21.15.83:0/2497886097 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f79b8001da0 con 0x7f79c01430c0 2024-03-25T19:23:56.503 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.503+0000 7f79a77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:56.503 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.503+0000 7f79a77fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:23:56.503 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.503+0000 7f79a77fe640 10 monclient(hunting): dump: 2024-03-25T19:23:56.503 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:23:56.503 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:23:56.503 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:56.503 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:56.503 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:56.503 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:23:56.503 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:23:56.503 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:23:56.503 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:23:56.503 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:56.507 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.506+0000 7f79a77fe640 1 -- 172.21.15.83:0/2497886097 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f79b00018f0 con 0x7f79c00a98f0 2024-03-25T19:23:56.507 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.506+0000 7f79a77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:56.507 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.506+0000 7f79a77fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:23:56.507 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.506+0000 7f79a77fe640 10 monclient(hunting): dump: 2024-03-25T19:23:56.507 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:23:56.507 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:23:56.507 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:56.507 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:56.507 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:56.507 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:23:56.507 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:23:56.507 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:23:56.507 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:23:56.507 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:56.508 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.506+0000 7f79a77fe640 1 -- 172.21.15.83:0/2497886097 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 194390765 0 0) 0x7f79c00b1770 con 0x7f79c00a98f0 2024-03-25T19:23:56.508 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.506+0000 7f79a77fe640 1 -- 172.21.15.83:0/2497886097 >> v1:172.21.15.103:6789/0 conn(0x7f79c01430c0 legacy=0x7f79c01342d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:56.508 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.506+0000 7f79a77fe640 1 -- 172.21.15.83:0/2497886097 >> v1:172.21.15.83:6789/0 conn(0x7f79c013f710 legacy=0x7f79c0133bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:56.508 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.506+0000 7f79a77fe640 10 monclient: _finish_hunting 0 2024-03-25T19:23:56.508 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.506+0000 7f79a77fe640 1 monclient: found mon.c 2024-03-25T19:23:56.508 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.506+0000 7f79a77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:56.508 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.506+0000 7f79a77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:23:56.508 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.506+0000 7f79a77fe640 1 -- 172.21.15.83:0/2497886097 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79c0138b00 con 0x7f79c00a98f0 2024-03-25T19:23:56.508 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.506+0000 7f79a77fe640 10 monclient: _finish_auth 0 2024-03-25T19:23:56.508 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.506+0000 7f79a77fe640 10 monclient: _check_auth_tickets 2024-03-25T19:23:56.508 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.506+0000 7f79a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:56.508 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.506+0000 7f79c5463640 5 monclient: authenticate success, global_id 4424 2024-03-25T19:23:56.508 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.506+0000 7f79c5463640 10 monclient: _renew_subs 2024-03-25T19:23:56.508 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.506+0000 7f79c5463640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:23:56.508 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.506+0000 7f79c5463640 1 -- 172.21.15.83:0/2497886097 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79c0139950 con 0x7f79c00a98f0 2024-03-25T19:23:56.509 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.506+0000 7f79c5463640 10 client.4424.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:23:56.509 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.506+0000 7f79c5463640 10 monclient: _renew_subs 2024-03-25T19:23:56.509 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.506+0000 7f79c5463640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:23:56.509 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.506+0000 7f79c5463640 1 -- 172.21.15.83:0/2497886097 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f79c0138230 con 0x7f79c00a98f0 2024-03-25T19:23:56.509 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.506+0000 7f79c5463640 1 librados: init done 2024-03-25T19:23:56.509 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.507+0000 7f79c5463640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:56.509 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.507+0000 7f79c5463640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:56.509 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.507+0000 7f79c5463640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:23:56.509 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.507+0000 7f79c5463640 1 -- 172.21.15.83:0/2497886097 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79c0074cc0 con 0x7f79c00a98f0 2024-03-25T19:23:56.814 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.814+0000 7f79a77fe640 1 -- 172.21.15.83:0/2497886097 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f79b00047e0 con 0x7f79c00a98f0 2024-03-25T19:23:56.815 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.814+0000 7f79a77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:56.815 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.815+0000 7f79a77fe640 1 -- 172.21.15.83:0/2497886097 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f79b0001d80 con 0x7f79c00a98f0 2024-03-25T19:23:56.815 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.815+0000 7f79a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:56.815 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.815+0000 7f79a77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:23:56.815 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.815+0000 7f79a77fe640 10 monclient: dump: 2024-03-25T19:23:56.815 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:23:56.815 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:23:56.815 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:56.815 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:23:56.815 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:56.815 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:23:56.815 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:23:56.815 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:23:56.815 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:23:56.815 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:23:56.816 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.815+0000 7f79bd7fa640 4 set_mon_vals no callback set 2024-03-25T19:23:56.816 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.815+0000 7f79a77fe640 1 -- 172.21.15.83:0/2497886097 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7f79b0004c30 con 0x7f79c00a98f0 2024-03-25T19:23:56.816 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.816+0000 7f79be7fc640 10 monclient: get_auth_request con 0x7f79ac06e170 auth_method 0 2024-03-25T19:23:56.816 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.816+0000 7f79a77fe640 10 client.4424.objecter ms_handle_connect 0x7f79ac06e170 2024-03-25T19:23:56.823 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.823+0000 7f79a77fe640 1 -- 172.21.15.83:0/2497886097 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7f79b0054720 con 0x7f79c00a98f0 2024-03-25T19:23:56.823 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.824+0000 7f79a77fe640 10 client.4424.objecter ms_dispatch 0x7f79c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:23:56.823 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.824+0000 7f79a77fe640 3 client.4424.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:23:56.824 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.824+0000 7f79a77fe640 3 client.4424.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:23:56.824 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.824+0000 7f79a77fe640 20 client.4424.objecter dump_active .. 0 homeless 2024-03-25T19:23:56.824 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.824+0000 7f79a77fe640 1 -- 172.21.15.83:0/2497886097 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f79b0050230 con 0x7f79c00a98f0 2024-03-25T19:23:56.824 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.824+0000 7f79a77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:56.824 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:56.824+0000 7f79a77fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:23:57.075 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:57.075+0000 7f79c5463640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:57.075 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:57.075+0000 7f79c5463640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:57.075 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:57.075+0000 7f79c5463640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:23:57.075 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:57.075+0000 7f79c5463640 1 -- 172.21.15.83:0/2497886097 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f79c014f2d0 con 0x7f79c00a98f0 2024-03-25T19:23:57.101 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:57.101+0000 7f79a77fe640 1 -- 172.21.15.83:0/2497886097 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7f79b0052160 con 0x7f79c00a98f0 2024-03-25T19:23:57.101 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:57.101+0000 7f79a77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:57.101 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:57.101+0000 7f79a77fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:23:57.101 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:23:57.101 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:23:57.104 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:57.103+0000 7f79c5463640 10 librados: watch_flush enter 2024-03-25T19:23:57.104 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:57.103+0000 7f79c5463640 10 librados: watch_flush exit 2024-03-25T19:23:57.104 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:57.103+0000 7f79c5463640 20 client.4424.objecter shutdown clearing up homeless session... 2024-03-25T19:23:57.104 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:57.103+0000 7f79c5463640 10 client.4424.objecter successfully canceled tick 2024-03-25T19:23:57.104 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:57.103+0000 7f79c5463640 1 -- 172.21.15.83:0/2497886097 >> v1:172.21.15.103:6800/345072327 conn(0x7f79ac06e170 legacy=0x7f79ac070630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:57.104 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:57.103+0000 7f79c5463640 10 monclient: shutdown 2024-03-25T19:23:57.104 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:57.103+0000 7f79c5463640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:57.104 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:57.103+0000 7f79c5463640 1 -- 172.21.15.83:0/2497886097 >> v1:172.21.15.83:6790/0 conn(0x7f79c00a98f0 legacy=0x7f79c01da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:57.104 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:57.104+0000 7f79c5463640 1 -- 172.21.15.83:0/2497886097 shutdown_connections 2024-03-25T19:23:57.104 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:57.104+0000 7f79c5463640 1 -- 172.21.15.83:0/2497886097 >> 172.21.15.83:0/2497886097 conn(0x7f79c00b7ea0 msgr2=0x7f79c0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:57.104 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:57.104+0000 7f79c5463640 1 -- 172.21.15.83:0/2497886097 shutdown_connections 2024-03-25T19:23:57.104 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:57.104+0000 7f79c5463640 1 -- 172.21.15.83:0/2497886097 wait complete. 2024-03-25T19:23:57.104 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:23:57.104+0000 7f79c5463640 1 librados: shutdown 2024-03-25T19:23:57.117 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:24:02.873 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:04:44.800 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:02.931 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:04:44.859 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:02.969 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:04:44.896 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:02.981 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:04:44.892 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:03.026 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:04:44.913 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:03.026 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:04:44.926 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:03.026 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:04:44.897 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:03.026 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:04:44.924 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:03.030 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:04:44.957 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:03.923 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:24:04.021 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.021+0000 7f2d3020e640 10 monclient: get_monmap_and_config 2024-03-25T19:24:04.021 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.021+0000 7f2d3020e640 10 monclient: build_initial_monmap 2024-03-25T19:24:04.021 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.021+0000 7f2d3020e640 10 monclient: monmap: 2024-03-25T19:24:04.021 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:24:04.021 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:24:04.021 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:24:04.022303+0000 2024-03-25T19:24:04.021 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:24:04.022303+0000 2024-03-25T19:24:04.021 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:04.021 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:24:04.021 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:24:04.021 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:24:04.021 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:24:04.021 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:04.022 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.021+0000 7f2d3020e640 1 Processor -- start 2024-03-25T19:24:04.023 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.022+0000 7f2d3020e640 1 -- start start 2024-03-25T19:24:04.023 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.022+0000 7f2d3020e640 10 monclient: init 2024-03-25T19:24:04.023 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.023+0000 7f2d3020e640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:04.023 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.023+0000 7f2d3020e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:24:04.023 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.023+0000 7f2d3020e640 10 monclient(hunting): picked mon.noname-c con 0x7f2d280a98f0 addr v1:172.21.15.103:6789/0 2024-03-25T19:24:04.023 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.023+0000 7f2d3020e640 10 monclient(hunting): picked mon.noname-b con 0x7f2d2813f710 addr v1:172.21.15.83:6790/0 2024-03-25T19:24:04.023 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.023+0000 7f2d3020e640 10 monclient(hunting): picked mon.noname-a con 0x7f2d281430c0 addr v1:172.21.15.83:6789/0 2024-03-25T19:24:04.023 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.023+0000 7f2d3020e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d28142520 con 0x7f2d281430c0 2024-03-25T19:24:04.023 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.023+0000 7f2d3020e640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d280b28a0 con 0x7f2d280a98f0 2024-03-25T19:24:04.023 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.023+0000 7f2d3020e640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d280b2a30 con 0x7f2d2813f710 2024-03-25T19:24:04.023 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.023+0000 7f2d3020e640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:04.023 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.023+0000 7f2d3020e640 10 monclient(hunting): authenticate will time out at 6301.294860s 2024-03-25T19:24:04.025 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.025+0000 7f2d2dfaa640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f2d280a98f0 0x7f2d280a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:35192/0 (socket says 172.21.15.83:35192) 2024-03-25T19:24:04.025 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.025+0000 7f2d2dfaa640 1 -- 172.21.15.83:0/2142005773 learned_addr learned my addr 172.21.15.83:0/2142005773 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:24:04.064 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.065+0000 7f2d2cfa8640 1 -- 172.21.15.83:0/2142005773 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2244867666 0 0) 0x7f2d28142520 con 0x7f2d281430c0 2024-03-25T19:24:04.064 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.065+0000 7f2d2cfa8640 10 monclient(hunting): _negotiate 2024-03-25T19:24:04.065 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.065+0000 7f2d2cfa8640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:04.065 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.065+0000 7f2d2cfa8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:04.065 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.065+0000 7f2d2cfa8640 10 monclient(hunting): my global_id is 4413 2024-03-25T19:24:04.065 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.065+0000 7f2d2cfa8640 1 -- 172.21.15.83:0/2142005773 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d14003520 con 0x7f2d281430c0 2024-03-25T19:24:04.138 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.138+0000 7f2d2cfa8640 1 -- 172.21.15.83:0/2142005773 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4140649981 0 0) 0x7f2d280b2a30 con 0x7f2d2813f710 2024-03-25T19:24:04.138 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.138+0000 7f2d2cfa8640 10 monclient(hunting): _negotiate 2024-03-25T19:24:04.138 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.138+0000 7f2d2cfa8640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:04.138 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.138+0000 7f2d2cfa8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:04.138 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.138+0000 7f2d2cfa8640 10 monclient(hunting): my global_id is 4427 2024-03-25T19:24:04.138 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.138+0000 7f2d2cfa8640 1 -- 172.21.15.83:0/2142005773 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d28142520 con 0x7f2d2813f710 2024-03-25T19:24:04.142 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.142+0000 7f2d2cfa8640 1 -- 172.21.15.83:0/2142005773 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1008317205 0 0) 0x7f2d280b28a0 con 0x7f2d280a98f0 2024-03-25T19:24:04.142 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.142+0000 7f2d2cfa8640 10 monclient(hunting): _negotiate 2024-03-25T19:24:04.142 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.142+0000 7f2d2cfa8640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:04.142 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.142+0000 7f2d2cfa8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:04.142 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.142+0000 7f2d2cfa8640 10 monclient(hunting): my global_id is 4420 2024-03-25T19:24:04.142 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.142+0000 7f2d2cfa8640 1 -- 172.21.15.83:0/2142005773 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d280b2a30 con 0x7f2d280a98f0 2024-03-25T19:24:04.150 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.150+0000 7f2d2cfa8640 1 -- 172.21.15.83:0/2142005773 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3090710898 0 0) 0x7f2d14003520 con 0x7f2d281430c0 2024-03-25T19:24:04.150 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.150+0000 7f2d2cfa8640 1 -- 172.21.15.83:0/2142005773 >> v1:172.21.15.83:6790/0 conn(0x7f2d2813f710 legacy=0x7f2d281429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:04.150 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.150+0000 7f2d2cfa8640 1 -- 172.21.15.83:0/2142005773 >> v1:172.21.15.103:6789/0 conn(0x7f2d280a98f0 legacy=0x7f2d280a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:04.150 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.150+0000 7f2d2cfa8640 10 monclient: _finish_hunting 0 2024-03-25T19:24:04.150 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.150+0000 7f2d2cfa8640 1 monclient: found mon.noname-a 2024-03-25T19:24:04.150 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.150+0000 7f2d2cfa8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:04.150 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.150+0000 7f2d2cfa8640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.83:6789/0 2024-03-25T19:24:04.150 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.150+0000 7f2d2cfa8640 1 -- 172.21.15.83:0/2142005773 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d280b44c0 con 0x7f2d281430c0 2024-03-25T19:24:04.150 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.150+0000 7f2d2cfa8640 10 monclient: _finish_auth 0 2024-03-25T19:24:04.150 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.150+0000 7f2d2cfa8640 10 monclient: _check_auth_tickets 2024-03-25T19:24:04.150 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.150+0000 7f2d2cfa8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:04.150 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.150+0000 7f2d2cfa8640 1 -- 172.21.15.83:0/2142005773 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f2d20001a10 con 0x7f2d281430c0 2024-03-25T19:24:04.151 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.150+0000 7f2d3020e640 5 monclient: authenticate success, global_id 4413 2024-03-25T19:24:04.151 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.150+0000 7f2d3020e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:24:04.151 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.150+0000 7f2d2cfa8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:04.151 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.150+0000 7f2d2cfa8640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:24:04.151 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.150+0000 7f2d2cfa8640 10 monclient: dump: 2024-03-25T19:24:04.151 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:24:04.151 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:24:04.151 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:04.151 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:04.151 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:04.151 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:24:04.151 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:24:04.151 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:24:04.151 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:24:04.151 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:04.151 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.150+0000 7f2d3020e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:24:04.172 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.172+0000 7f2d2cfa8640 1 -- 172.21.15.83:0/2142005773 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d20001b70 con 0x7f2d281430c0 2024-03-25T19:24:04.172 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.172+0000 7f2d2cfa8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:04.172 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.172+0000 7f2d2cfa8640 1 -- 172.21.15.83:0/2142005773 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f2d20001f80 con 0x7f2d281430c0 2024-03-25T19:24:04.172 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.172+0000 7f2d3020e640 10 monclient: get_monmap_and_config success 2024-03-25T19:24:04.172 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.172+0000 7f2d3020e640 4 set_mon_vals no callback set 2024-03-25T19:24:04.172 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.172+0000 7f2d2cfa8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:04.172 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.172+0000 7f2d2cfa8640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:24:04.172 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.172+0000 7f2d2cfa8640 10 monclient: dump: 2024-03-25T19:24:04.172 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:24:04.173 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:24:04.173 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:04.173 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:04.173 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:04.173 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:24:04.173 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:24:04.173 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:24:04.173 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:24:04.173 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:04.175 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.173+0000 7f2d3020e640 10 monclient: shutdown 2024-03-25T19:24:04.175 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.173+0000 7f2d3020e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:04.175 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.173+0000 7f2d3020e640 1 -- 172.21.15.83:0/2142005773 >> v1:172.21.15.83:6789/0 conn(0x7f2d281430c0 legacy=0x7f2d2814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:04.175 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.173+0000 7f2d3020e640 1 -- 172.21.15.83:0/2142005773 shutdown_connections 2024-03-25T19:24:04.175 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.173+0000 7f2d3020e640 1 -- 172.21.15.83:0/2142005773 >> 172.21.15.83:0/2142005773 conn(0x7f2d280b7ea0 msgr2=0x7f2d280b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:04.176 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.173+0000 7f2d3020e640 1 -- 172.21.15.83:0/2142005773 shutdown_connections 2024-03-25T19:24:04.176 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.173+0000 7f2d3020e640 1 -- 172.21.15.83:0/2142005773 wait complete. 2024-03-25T19:24:04.176 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.174+0000 7f2d3020e640 10 monclient: build_initial_monmap 2024-03-25T19:24:04.176 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.174+0000 7f2d3020e640 10 monclient: monmap: 2024-03-25T19:24:04.176 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:24:04.176 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:24:04.176 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:24:04.176 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:24:04.176 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:04.176 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:24:04.176 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:24:04.176 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:24:04.176 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:24:04.176 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:04.176 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.174+0000 7f2d3020e640 1 librados: starting msgr at 2024-03-25T19:24:04.176 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.174+0000 7f2d3020e640 1 librados: starting objecter 2024-03-25T19:24:04.176 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.174+0000 7f2d3020e640 1 Processor -- start 2024-03-25T19:24:04.177 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.175+0000 7f2d3020e640 1 -- start start 2024-03-25T19:24:04.177 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.175+0000 7f2d3020e640 1 librados: setting wanted keys 2024-03-25T19:24:04.177 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.175+0000 7f2d3020e640 1 librados: calling monclient init 2024-03-25T19:24:04.177 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.175+0000 7f2d3020e640 10 monclient: init 2024-03-25T19:24:04.177 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.175+0000 7f2d3020e640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:04.177 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.175+0000 7f2d3020e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:24:04.177 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.175+0000 7f2d3020e640 10 monclient(hunting): picked mon.noname-b con 0x7f2d280a98f0 addr v1:172.21.15.103:6789/0 2024-03-25T19:24:04.177 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.175+0000 7f2d3020e640 10 monclient(hunting): picked mon.noname-c con 0x7f2d2813f710 addr v1:172.21.15.83:6790/0 2024-03-25T19:24:04.177 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.175+0000 7f2d3020e640 10 monclient(hunting): picked mon.noname-a con 0x7f2d281430c0 addr v1:172.21.15.83:6789/0 2024-03-25T19:24:04.177 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.175+0000 7f2d3020e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d280abb00 con 0x7f2d281430c0 2024-03-25T19:24:04.177 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.175+0000 7f2d3020e640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d280abde0 con 0x7f2d280a98f0 2024-03-25T19:24:04.177 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.175+0000 7f2d3020e640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d28138e20 con 0x7f2d2813f710 2024-03-25T19:24:04.177 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.175+0000 7f2d3020e640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:04.177 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.175+0000 7f2d3020e640 10 monclient(hunting): authenticate will time out at 6301.447539s 2024-03-25T19:24:04.178 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.178+0000 7f2d2dfaa640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f2d280a98f0 0x7f2d281385e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:35256/0 (socket says 172.21.15.83:35256) 2024-03-25T19:24:04.178 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.178+0000 7f2d2dfaa640 1 -- 172.21.15.83:0/3162722416 learned_addr learned my addr 172.21.15.83:0/3162722416 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:24:04.178 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.179+0000 7f2d1e7fc640 10 client.?.objecter ms_handle_connect 0x7f2d280a98f0 2024-03-25T19:24:04.178 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.179+0000 7f2d1e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:04.185 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.185+0000 7f2d1e7fc640 1 -- 172.21.15.83:0/3162722416 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 886267762 0 0) 0x7f2d280abde0 con 0x7f2d280a98f0 2024-03-25T19:24:04.185 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.185+0000 7f2d1e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:24:04.185 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.185+0000 7f2d1e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:04.185 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.185+0000 7f2d1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:04.185 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.185+0000 7f2d1e7fc640 10 monclient(hunting): my global_id is 4423 2024-03-25T19:24:04.185 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.185+0000 7f2d1e7fc640 1 -- 172.21.15.83:0/3162722416 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d0c004eb0 con 0x7f2d280a98f0 2024-03-25T19:24:04.190 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.190+0000 7f2d1e7fc640 10 client.?.objecter ms_handle_connect 0x7f2d2813f710 2024-03-25T19:24:04.190 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.190+0000 7f2d1e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:04.191 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.191+0000 7f2d1e7fc640 10 client.?.objecter ms_handle_connect 0x7f2d281430c0 2024-03-25T19:24:04.191 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.191+0000 7f2d1e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:04.193 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.193+0000 7f2d1e7fc640 1 -- 172.21.15.83:0/3162722416 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 934305108 0 0) 0x7f2d0c004eb0 con 0x7f2d280a98f0 2024-03-25T19:24:04.193 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.193+0000 7f2d1e7fc640 1 -- 172.21.15.83:0/3162722416 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d280abde0 con 0x7f2d280a98f0 2024-03-25T19:24:04.195 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.195+0000 7f2d1e7fc640 1 -- 172.21.15.83:0/3162722416 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f2d24001a30 con 0x7f2d280a98f0 2024-03-25T19:24:04.195 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.195+0000 7f2d1e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:04.195 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.195+0000 7f2d1e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:24:04.195 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.195+0000 7f2d1e7fc640 10 monclient(hunting): dump: 2024-03-25T19:24:04.195 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:24:04.195 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:24:04.195 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:04.195 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:04.195 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:04.195 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:24:04.196 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:24:04.196 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:24:04.196 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:24:04.196 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:04.196 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.195+0000 7f2d1e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:24:04.201 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.201+0000 7f2d1e7fc640 1 -- 172.21.15.83:0/3162722416 <== mon.1 v1:172.21.15.103:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2243616882 0 0) 0x7f2d280abde0 con 0x7f2d280a98f0 2024-03-25T19:24:04.201 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.201+0000 7f2d1e7fc640 1 -- 172.21.15.83:0/3162722416 >> v1:172.21.15.83:6790/0 conn(0x7f2d2813f710 legacy=0x7f2d28136330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:04.201 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.201+0000 7f2d1e7fc640 1 -- 172.21.15.83:0/3162722416 >> v1:172.21.15.83:6789/0 conn(0x7f2d281430c0 legacy=0x7f2d280ac200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:04.201 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.201+0000 7f2d1e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:24:04.201 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.201+0000 7f2d1e7fc640 1 monclient: found mon.b 2024-03-25T19:24:04.201 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.201+0000 7f2d1e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:04.201 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.201+0000 7f2d1e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:24:04.201 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.201+0000 7f2d1e7fc640 1 -- 172.21.15.83:0/3162722416 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d28137b00 con 0x7f2d280a98f0 2024-03-25T19:24:04.202 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.201+0000 7f2d1e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:24:04.202 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.201+0000 7f2d1e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:24:04.202 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.201+0000 7f2d1e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:04.202 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.201+0000 7f2d3020e640 5 monclient: authenticate success, global_id 4423 2024-03-25T19:24:04.202 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.201+0000 7f2d3020e640 10 monclient: _renew_subs 2024-03-25T19:24:04.202 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.201+0000 7f2d3020e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:24:04.202 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.201+0000 7f2d3020e640 1 -- 172.21.15.83:0/3162722416 --> v1:172.21.15.103:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d281e34b0 con 0x7f2d280a98f0 2024-03-25T19:24:04.202 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.201+0000 7f2d3020e640 10 client.4423.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:24:04.202 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.201+0000 7f2d3020e640 10 monclient: _renew_subs 2024-03-25T19:24:04.202 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.201+0000 7f2d3020e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:24:04.202 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.201+0000 7f2d3020e640 1 -- 172.21.15.83:0/3162722416 --> v1:172.21.15.103:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d281e3820 con 0x7f2d280a98f0 2024-03-25T19:24:04.202 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.201+0000 7f2d3020e640 1 librados: init done 2024-03-25T19:24:04.203 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.202+0000 7f2d3020e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:04.203 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.202+0000 7f2d3020e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:04.203 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.202+0000 7f2d3020e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:24:04.203 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.202+0000 7f2d3020e640 1 -- 172.21.15.83:0/3162722416 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d28138fb0 con 0x7f2d280a98f0 2024-03-25T19:24:04.500 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.500+0000 7f2d1e7fc640 1 -- 172.21.15.83:0/3162722416 <== mon.1 v1:172.21.15.103:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d24001b90 con 0x7f2d280a98f0 2024-03-25T19:24:04.500 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.500+0000 7f2d1e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:04.500 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.500+0000 7f2d1e7fc640 1 -- 172.21.15.83:0/3162722416 <== mon.1 v1:172.21.15.103:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f2d24004cb0 con 0x7f2d280a98f0 2024-03-25T19:24:04.500 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.500+0000 7f2d1e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:04.500 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.500+0000 7f2d1e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:24:04.500 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.500+0000 7f2d1e7fc640 10 monclient: dump: 2024-03-25T19:24:04.501 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:24:04.501 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:24:04.501 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:04.501 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:04.501 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:04.501 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:24:04.501 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:24:04.501 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:24:04.501 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:24:04.501 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:04.501 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.500+0000 7f2d1ffff640 4 set_mon_vals no callback set 2024-03-25T19:24:04.502 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.501+0000 7f2d1e7fc640 1 -- 172.21.15.83:0/3162722416 <== mon.1 v1:172.21.15.103:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7f2d24004ed0 con 0x7f2d280a98f0 2024-03-25T19:24:04.502 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.502+0000 7f2d2d7a9640 10 monclient: get_auth_request con 0x7f2d0c06dca0 auth_method 0 2024-03-25T19:24:04.502 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.502+0000 7f2d1e7fc640 10 client.4423.objecter ms_handle_connect 0x7f2d0c06dca0 2024-03-25T19:24:04.505 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.506+0000 7f2d1e7fc640 1 -- 172.21.15.83:0/3162722416 <== mon.1 v1:172.21.15.103:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7f2d24054960 con 0x7f2d280a98f0 2024-03-25T19:24:04.505 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.506+0000 7f2d1e7fc640 10 client.4423.objecter ms_dispatch 0x7f2d28000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:24:04.505 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.506+0000 7f2d1e7fc640 3 client.4423.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:24:04.506 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.506+0000 7f2d1e7fc640 3 client.4423.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:24:04.506 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.506+0000 7f2d1e7fc640 20 client.4423.objecter dump_active .. 0 homeless 2024-03-25T19:24:04.506 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.506+0000 7f2d1e7fc640 1 -- 172.21.15.83:0/3162722416 <== mon.1 v1:172.21.15.103:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2d24050470 con 0x7f2d280a98f0 2024-03-25T19:24:04.506 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.506+0000 7f2d1e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:04.506 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.506+0000 7f2d1e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:24:04.748 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.748+0000 7f2d3020e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:04.748 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.748+0000 7f2d3020e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:04.748 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.748+0000 7f2d3020e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:24:04.748 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.748+0000 7f2d3020e640 1 -- 172.21.15.83:0/3162722416 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2d280627b0 con 0x7f2d280a98f0 2024-03-25T19:24:04.768 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.768+0000 7f2d1e7fc640 1 -- 172.21.15.83:0/3162722416 <== mon.1 v1:172.21.15.103:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7f2d240523a0 con 0x7f2d280a98f0 2024-03-25T19:24:04.768 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.768+0000 7f2d1e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:04.768 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.768+0000 7f2d1e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:24:04.768 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:24:04.768 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:24:04.770 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.770+0000 7f2d3020e640 10 librados: watch_flush enter 2024-03-25T19:24:04.771 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.770+0000 7f2d3020e640 10 librados: watch_flush exit 2024-03-25T19:24:04.771 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.770+0000 7f2d3020e640 20 client.4423.objecter shutdown clearing up homeless session... 2024-03-25T19:24:04.771 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.770+0000 7f2d3020e640 10 client.4423.objecter successfully canceled tick 2024-03-25T19:24:04.771 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.770+0000 7f2d3020e640 1 -- 172.21.15.83:0/3162722416 >> v1:172.21.15.103:6800/345072327 conn(0x7f2d0c06dca0 legacy=0x7f2d0c070160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:04.771 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.770+0000 7f2d3020e640 10 monclient: shutdown 2024-03-25T19:24:04.771 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.770+0000 7f2d3020e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:04.771 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.770+0000 7f2d3020e640 1 -- 172.21.15.83:0/3162722416 >> v1:172.21.15.103:6789/0 conn(0x7f2d280a98f0 legacy=0x7f2d281385e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:04.771 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.770+0000 7f2d3020e640 1 -- 172.21.15.83:0/3162722416 shutdown_connections 2024-03-25T19:24:04.771 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.770+0000 7f2d3020e640 1 -- 172.21.15.83:0/3162722416 >> 172.21.15.83:0/3162722416 conn(0x7f2d280b7ea0 msgr2=0x7f2d2814ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:04.771 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.770+0000 7f2d3020e640 1 -- 172.21.15.83:0/3162722416 shutdown_connections 2024-03-25T19:24:04.771 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.770+0000 7f2d3020e640 1 -- 172.21.15.83:0/3162722416 wait complete. 2024-03-25T19:24:04.771 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:04.771+0000 7f2d3020e640 1 librados: shutdown 2024-03-25T19:24:04.783 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:24:11.590 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:24:11.713 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.713+0000 7f6171ced640 10 monclient: get_monmap_and_config 2024-03-25T19:24:11.713 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.713+0000 7f6171ced640 10 monclient: build_initial_monmap 2024-03-25T19:24:11.713 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.713+0000 7f6171ced640 10 monclient: monmap: 2024-03-25T19:24:11.713 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:24:11.713 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:24:11.713 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:24:11.714390+0000 2024-03-25T19:24:11.713 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:24:11.714390+0000 2024-03-25T19:24:11.713 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:11.713 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:24:11.713 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:24:11.713 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:24:11.714 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:24:11.714 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:11.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.714+0000 7f6171ced640 1 Processor -- start 2024-03-25T19:24:11.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.714+0000 7f6171ced640 1 -- start start 2024-03-25T19:24:11.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.714+0000 7f6171ced640 10 monclient: init 2024-03-25T19:24:11.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.715+0000 7f6171ced640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:11.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.715+0000 7f6171ced640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:24:11.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.715+0000 7f6171ced640 10 monclient(hunting): picked mon.noname-c con 0x7f616c06ab80 addr v1:172.21.15.103:6789/0 2024-03-25T19:24:11.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.715+0000 7f6171ced640 10 monclient(hunting): picked mon.noname-a con 0x7f616c06af60 addr v1:172.21.15.83:6789/0 2024-03-25T19:24:11.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.715+0000 7f6171ced640 10 monclient(hunting): picked mon.noname-b con 0x7f616c072790 addr v1:172.21.15.83:6790/0 2024-03-25T19:24:11.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.715+0000 7f6171ced640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f616c142ad0 con 0x7f616c06af60 2024-03-25T19:24:11.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.715+0000 7f6171ced640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f616c13ffa0 con 0x7f616c06ab80 2024-03-25T19:24:11.716 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.715+0000 7f6171ced640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f616c0b1130 con 0x7f616c072790 2024-03-25T19:24:11.716 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.715+0000 7f6171ced640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:11.716 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.715+0000 7f6171ced640 10 monclient(hunting): authenticate will time out at 6308.987176s 2024-03-25T19:24:11.725 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.725+0000 7f616affd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f616c06af60 0x7f616c13f890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:58742/0 (socket says 172.21.15.83:58742) 2024-03-25T19:24:11.725 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.725+0000 7f616affd640 1 -- 172.21.15.83:0/4224673252 learned_addr learned my addr 172.21.15.83:0/4224673252 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:24:11.736 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.736+0000 7f616a7fc640 1 -- 172.21.15.83:0/4224673252 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2706246290 0 0) 0x7f616c0b1130 con 0x7f616c072790 2024-03-25T19:24:11.736 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.736+0000 7f616a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:24:11.736 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.736+0000 7f616a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:11.736 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.736+0000 7f616a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:11.736 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.736+0000 7f616a7fc640 10 monclient(hunting): my global_id is 4433 2024-03-25T19:24:11.736 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.736+0000 7f616a7fc640 1 -- 172.21.15.83:0/4224673252 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6154003520 con 0x7f616c072790 2024-03-25T19:24:11.742 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.742+0000 7f616a7fc640 1 -- 172.21.15.83:0/4224673252 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2250453521 0 0) 0x7f616c142ad0 con 0x7f616c06af60 2024-03-25T19:24:11.742 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.742+0000 7f616a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:24:11.742 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.742+0000 7f616a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:11.742 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.742+0000 7f616a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:11.742 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.742+0000 7f616a7fc640 10 monclient(hunting): my global_id is 4419 2024-03-25T19:24:11.742 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.742+0000 7f616a7fc640 1 -- 172.21.15.83:0/4224673252 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f616c0b1130 con 0x7f616c06af60 2024-03-25T19:24:11.744 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.744+0000 7f616a7fc640 1 -- 172.21.15.83:0/4224673252 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4128581317 0 0) 0x7f616c13ffa0 con 0x7f616c06ab80 2024-03-25T19:24:11.744 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.744+0000 7f616a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:24:11.744 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.744+0000 7f616a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:11.744 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.744+0000 7f616a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:11.744 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.744+0000 7f616a7fc640 10 monclient(hunting): my global_id is 4426 2024-03-25T19:24:11.744 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.744+0000 7f616a7fc640 1 -- 172.21.15.83:0/4224673252 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f616c142ad0 con 0x7f616c06ab80 2024-03-25T19:24:11.756 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.756+0000 7f616a7fc640 1 -- 172.21.15.83:0/4224673252 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 589608618 0 0) 0x7f616c142ad0 con 0x7f616c06ab80 2024-03-25T19:24:11.756 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.756+0000 7f616a7fc640 1 -- 172.21.15.83:0/4224673252 >> v1:172.21.15.83:6790/0 conn(0x7f616c072790 legacy=0x7f616c14dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:11.756 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.756+0000 7f616a7fc640 1 -- 172.21.15.83:0/4224673252 >> v1:172.21.15.83:6789/0 conn(0x7f616c06af60 legacy=0x7f616c13f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:11.756 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.756+0000 7f616a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:24:11.756 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.756+0000 7f616a7fc640 1 monclient: found mon.noname-c 2024-03-25T19:24:11.756 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.756+0000 7f616a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:11.756 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.756+0000 7f616a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.103:6789/0 2024-03-25T19:24:11.756 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.756+0000 7f616a7fc640 1 -- 172.21.15.83:0/4224673252 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f616c0af520 con 0x7f616c06ab80 2024-03-25T19:24:11.756 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.756+0000 7f616a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:24:11.756 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.756+0000 7f616a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:24:11.757 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.756+0000 7f616a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:11.757 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.756+0000 7f616a7fc640 1 -- 172.21.15.83:0/4224673252 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f615c0015b0 con 0x7f616c06ab80 2024-03-25T19:24:11.757 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.756+0000 7f6171ced640 5 monclient: authenticate success, global_id 4426 2024-03-25T19:24:11.757 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.756+0000 7f6171ced640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:24:11.757 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.756+0000 7f616a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:11.757 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.756+0000 7f616a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:24:11.757 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.756+0000 7f616a7fc640 10 monclient: dump: 2024-03-25T19:24:11.757 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:24:11.757 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:24:11.757 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:11.757 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:11.757 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:11.757 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:24:11.757 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:24:11.757 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:24:11.757 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:24:11.758 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:11.758 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.756+0000 7f6171ced640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:24:11.786 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.787+0000 7f616a7fc640 1 -- 172.21.15.83:0/4224673252 <== mon.1 v1:172.21.15.103:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f615c001710 con 0x7f616c06ab80 2024-03-25T19:24:11.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.787+0000 7f616a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:11.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.787+0000 7f616a7fc640 1 -- 172.21.15.83:0/4224673252 <== mon.1 v1:172.21.15.103:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f615c001b60 con 0x7f616c06ab80 2024-03-25T19:24:11.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.787+0000 7f616a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:11.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.787+0000 7f616a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:24:11.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.787+0000 7f616a7fc640 10 monclient: dump: 2024-03-25T19:24:11.787 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:24:11.787 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:24:11.787 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:11.787 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:11.787 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:11.787 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:24:11.787 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:24:11.787 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:24:11.787 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:24:11.788 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:11.788 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.787+0000 7f6171ced640 10 monclient: get_monmap_and_config success 2024-03-25T19:24:11.788 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.787+0000 7f6171ced640 4 set_mon_vals no callback set 2024-03-25T19:24:11.789 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.787+0000 7f6171ced640 10 monclient: shutdown 2024-03-25T19:24:11.789 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.787+0000 7f6171ced640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:11.789 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.787+0000 7f6171ced640 1 -- 172.21.15.83:0/4224673252 >> v1:172.21.15.103:6789/0 conn(0x7f616c06ab80 legacy=0x7f616c1423c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:11.789 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.788+0000 7f6171ced640 1 -- 172.21.15.83:0/4224673252 shutdown_connections 2024-03-25T19:24:11.789 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.788+0000 7f6171ced640 1 -- 172.21.15.83:0/4224673252 >> 172.21.15.83:0/4224673252 conn(0x7f616c0b4a80 msgr2=0x7f616c0b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:11.790 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.788+0000 7f6171ced640 1 -- 172.21.15.83:0/4224673252 shutdown_connections 2024-03-25T19:24:11.790 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.788+0000 7f6171ced640 1 -- 172.21.15.83:0/4224673252 wait complete. 2024-03-25T19:24:11.790 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.789+0000 7f6171ced640 10 monclient: build_initial_monmap 2024-03-25T19:24:11.790 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.789+0000 7f6171ced640 10 monclient: monmap: 2024-03-25T19:24:11.790 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:24:11.790 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:24:11.790 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:24:11.790 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:24:11.790 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:11.790 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:24:11.790 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:24:11.790 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:24:11.790 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:24:11.791 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:11.791 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.789+0000 7f6171ced640 1 librados: starting msgr at 2024-03-25T19:24:11.791 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.789+0000 7f6171ced640 1 librados: starting objecter 2024-03-25T19:24:11.791 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.789+0000 7f6171ced640 1 Processor -- start 2024-03-25T19:24:11.791 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.789+0000 7f6171ced640 1 -- start start 2024-03-25T19:24:11.791 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.789+0000 7f6171ced640 1 librados: setting wanted keys 2024-03-25T19:24:11.791 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.789+0000 7f6171ced640 1 librados: calling monclient init 2024-03-25T19:24:11.791 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.789+0000 7f6171ced640 10 monclient: init 2024-03-25T19:24:11.792 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.790+0000 7f6171ced640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:11.792 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.790+0000 7f6171ced640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:24:11.792 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.790+0000 7f6171ced640 10 monclient(hunting): picked mon.noname-a con 0x7f616c06ab80 addr v1:172.21.15.83:6789/0 2024-03-25T19:24:11.792 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.790+0000 7f6171ced640 10 monclient(hunting): picked mon.noname-b con 0x7f616c072790 addr v1:172.21.15.103:6789/0 2024-03-25T19:24:11.793 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.790+0000 7f6171ced640 10 monclient(hunting): picked mon.noname-c con 0x7f616c13b5d0 addr v1:172.21.15.83:6790/0 2024-03-25T19:24:11.793 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.790+0000 7f6171ced640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f616c1ce200 con 0x7f616c06ab80 2024-03-25T19:24:11.793 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.790+0000 7f6171ced640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f616c1cf010 con 0x7f616c072790 2024-03-25T19:24:11.793 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.790+0000 7f6171ced640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f616c1cf1a0 con 0x7f616c13b5d0 2024-03-25T19:24:11.793 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.790+0000 7f6171ced640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:11.793 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.790+0000 7f6171ced640 10 monclient(hunting): authenticate will time out at 6309.062707s 2024-03-25T19:24:11.799 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.799+0000 7f616b7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f616c06ab80 0x7f616c1ce900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:58794/0 (socket says 172.21.15.83:58794) 2024-03-25T19:24:11.799 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.799+0000 7f616b7fe640 1 -- 172.21.15.83:0/3653634678 learned_addr learned my addr 172.21.15.83:0/3653634678 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:24:11.802 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.802+0000 7f6153fff640 10 client.?.objecter ms_handle_connect 0x7f616c13b5d0 2024-03-25T19:24:11.802 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.802+0000 7f6153fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:11.805 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.805+0000 7f6153fff640 10 client.?.objecter ms_handle_connect 0x7f616c06ab80 2024-03-25T19:24:11.805 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.805+0000 7f6153fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:11.809 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.810+0000 7f6153fff640 1 -- 172.21.15.83:0/3653634678 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 119424922 0 0) 0x7f616c1cf1a0 con 0x7f616c13b5d0 2024-03-25T19:24:11.809 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.810+0000 7f6153fff640 10 monclient(hunting): _negotiate 2024-03-25T19:24:11.810 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.810+0000 7f6153fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:11.810 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.810+0000 7f6153fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:11.810 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.810+0000 7f6153fff640 10 monclient(hunting): my global_id is 4436 2024-03-25T19:24:11.810 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.810+0000 7f6153fff640 1 -- 172.21.15.83:0/3653634678 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6158004eb0 con 0x7f616c13b5d0 2024-03-25T19:24:11.811 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.812+0000 7f6153fff640 10 client.?.objecter ms_handle_connect 0x7f616c072790 2024-03-25T19:24:11.811 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.812+0000 7f6153fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:11.816 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.816+0000 7f6153fff640 1 -- 172.21.15.83:0/3653634678 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1033534937 0 0) 0x7f616c1cf010 con 0x7f616c072790 2024-03-25T19:24:11.816 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.817+0000 7f6153fff640 10 monclient(hunting): _negotiate 2024-03-25T19:24:11.816 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.817+0000 7f6153fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:11.816 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.817+0000 7f6153fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:11.816 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.817+0000 7f6153fff640 10 monclient(hunting): my global_id is 4429 2024-03-25T19:24:11.816 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.817+0000 7f6153fff640 1 -- 172.21.15.83:0/3653634678 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f616c1cf1a0 con 0x7f616c072790 2024-03-25T19:24:11.821 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.821+0000 7f6153fff640 1 -- 172.21.15.83:0/3653634678 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1175182904 0 0) 0x7f616c1ce200 con 0x7f616c06ab80 2024-03-25T19:24:11.821 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.821+0000 7f6153fff640 10 monclient(hunting): _negotiate 2024-03-25T19:24:11.821 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.821+0000 7f6153fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:11.821 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.821+0000 7f6153fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:11.821 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.821+0000 7f6153fff640 10 monclient(hunting): my global_id is 4422 2024-03-25T19:24:11.821 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.821+0000 7f6153fff640 1 -- 172.21.15.83:0/3653634678 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f616c1cf010 con 0x7f616c06ab80 2024-03-25T19:24:11.833 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.833+0000 7f6153fff640 1 -- 172.21.15.83:0/3653634678 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2545825685 0 0) 0x7f616c1cf1a0 con 0x7f616c072790 2024-03-25T19:24:11.833 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.833+0000 7f6153fff640 1 -- 172.21.15.83:0/3653634678 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f616c1ce200 con 0x7f616c072790 2024-03-25T19:24:11.833 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.833+0000 7f6153fff640 1 -- 172.21.15.83:0/3653634678 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f6160001930 con 0x7f616c072790 2024-03-25T19:24:11.833 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.833+0000 7f6153fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:11.833 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.833+0000 7f6153fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:24:11.833 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.833+0000 7f6153fff640 10 monclient(hunting): dump: 2024-03-25T19:24:11.833 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:24:11.833 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:24:11.833 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:11.833 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:11.833 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:11.833 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:24:11.833 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:24:11.834 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:24:11.834 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:24:11.834 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:11.834 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.833+0000 7f6153fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:24:11.837 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.836+0000 7f6153fff640 1 -- 172.21.15.83:0/3653634678 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2899191931 0 0) 0x7f6158004eb0 con 0x7f616c13b5d0 2024-03-25T19:24:11.837 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.836+0000 7f6153fff640 1 -- 172.21.15.83:0/3653634678 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f616c1cf1a0 con 0x7f616c13b5d0 2024-03-25T19:24:11.837 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.836+0000 7f6153fff640 1 -- 172.21.15.83:0/3653634678 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f6164001b30 con 0x7f616c13b5d0 2024-03-25T19:24:11.837 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.836+0000 7f6153fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:11.837 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.836+0000 7f6153fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:24:11.837 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.836+0000 7f6153fff640 10 monclient(hunting): dump: 2024-03-25T19:24:11.837 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:24:11.837 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:24:11.837 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:11.837 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:11.837 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:11.837 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:24:11.837 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:24:11.837 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:24:11.837 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:24:11.838 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:11.838 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.837+0000 7f6153fff640 1 -- 172.21.15.83:0/3653634678 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2826905079 0 0) 0x7f616c1cf010 con 0x7f616c06ab80 2024-03-25T19:24:11.838 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.837+0000 7f6153fff640 1 -- 172.21.15.83:0/3653634678 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6158004eb0 con 0x7f616c06ab80 2024-03-25T19:24:11.838 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.837+0000 7f6153fff640 1 -- 172.21.15.83:0/3653634678 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f615c001a60 con 0x7f616c06ab80 2024-03-25T19:24:11.838 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.837+0000 7f6153fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:11.838 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.837+0000 7f6153fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:24:11.838 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.837+0000 7f6153fff640 10 monclient(hunting): dump: 2024-03-25T19:24:11.838 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:24:11.838 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:24:11.838 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:11.838 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:11.838 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:11.838 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:24:11.838 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:24:11.838 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:24:11.838 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:24:11.839 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:11.851 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.851+0000 7f6153fff640 1 -- 172.21.15.83:0/3653634678 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2418462901 0 0) 0x7f6158004eb0 con 0x7f616c06ab80 2024-03-25T19:24:11.851 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.851+0000 7f6153fff640 1 -- 172.21.15.83:0/3653634678 >> v1:172.21.15.83:6790/0 conn(0x7f616c13b5d0 legacy=0x7f616c1cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:11.852 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.851+0000 7f6153fff640 1 -- 172.21.15.83:0/3653634678 >> v1:172.21.15.103:6789/0 conn(0x7f616c072790 legacy=0x7f616c13aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:11.852 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.851+0000 7f6153fff640 10 monclient: _finish_hunting 0 2024-03-25T19:24:11.852 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.851+0000 7f6153fff640 1 monclient: found mon.a 2024-03-25T19:24:11.852 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.851+0000 7f6153fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:11.852 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.851+0000 7f6153fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:24:11.852 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.851+0000 7f6153fff640 1 -- 172.21.15.83:0/3653634678 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f616c13ba00 con 0x7f616c06ab80 2024-03-25T19:24:11.852 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.851+0000 7f6153fff640 10 monclient: _finish_auth 0 2024-03-25T19:24:11.852 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.851+0000 7f6153fff640 10 monclient: _check_auth_tickets 2024-03-25T19:24:11.852 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.851+0000 7f6153fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:11.852 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.851+0000 7f6171ced640 5 monclient: authenticate success, global_id 4422 2024-03-25T19:24:11.852 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.851+0000 7f6171ced640 10 monclient: _renew_subs 2024-03-25T19:24:11.852 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.851+0000 7f6171ced640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:24:11.852 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.851+0000 7f6171ced640 1 -- 172.21.15.83:0/3653634678 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f616c13bc50 con 0x7f616c06ab80 2024-03-25T19:24:11.852 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.851+0000 7f6171ced640 10 client.4422.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:24:11.852 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.851+0000 7f6171ced640 10 monclient: _renew_subs 2024-03-25T19:24:11.853 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.851+0000 7f6171ced640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:24:11.853 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.851+0000 7f6171ced640 1 -- 172.21.15.83:0/3653634678 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f616c1cc770 con 0x7f616c06ab80 2024-03-25T19:24:11.853 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.851+0000 7f6171ced640 1 librados: init done 2024-03-25T19:24:11.853 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.852+0000 7f6171ced640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:11.853 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.852+0000 7f6171ced640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:11.853 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.852+0000 7f6171ced640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:24:11.854 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:11.852+0000 7f6171ced640 1 -- 172.21.15.83:0/3653634678 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f616c1cf330 con 0x7f616c06ab80 2024-03-25T19:24:12.161 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:12.161+0000 7f6153fff640 1 -- 172.21.15.83:0/3653634678 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f615c0038a0 con 0x7f616c06ab80 2024-03-25T19:24:12.162 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:12.161+0000 7f6153fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:12.162 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:12.161+0000 7f6153fff640 1 -- 172.21.15.83:0/3653634678 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f615c003cb0 con 0x7f616c06ab80 2024-03-25T19:24:12.162 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:12.161+0000 7f6153fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:12.162 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:12.161+0000 7f6153fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:24:12.162 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:12.161+0000 7f6153fff640 10 monclient: dump: 2024-03-25T19:24:12.162 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:24:12.162 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:24:12.162 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:12.162 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:12.162 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:12.162 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:24:12.162 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:24:12.162 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:24:12.162 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:24:12.163 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:12.163 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:12.161+0000 7f6169ffb640 4 set_mon_vals no callback set 2024-03-25T19:24:12.163 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:12.162+0000 7f6153fff640 1 -- 172.21.15.83:0/3653634678 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7f615c003ed0 con 0x7f616c06ab80 2024-03-25T19:24:12.163 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:12.163+0000 7f616affd640 10 monclient: get_auth_request con 0x7f615806e160 auth_method 0 2024-03-25T19:24:12.163 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:12.163+0000 7f6153fff640 10 client.4422.objecter ms_handle_connect 0x7f615806e160 2024-03-25T19:24:12.169 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:12.169+0000 7f6153fff640 1 -- 172.21.15.83:0/3653634678 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7f615c001bc0 con 0x7f616c06ab80 2024-03-25T19:24:12.169 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:12.169+0000 7f6153fff640 10 client.4422.objecter ms_dispatch 0x7f616c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:24:12.169 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:12.169+0000 7f6153fff640 3 client.4422.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:24:12.169 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:12.169+0000 7f6153fff640 3 client.4422.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:24:12.169 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:12.169+0000 7f6153fff640 20 client.4422.objecter dump_active .. 0 homeless 2024-03-25T19:24:12.169 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:12.169+0000 7f6153fff640 1 -- 172.21.15.83:0/3653634678 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f615c050350 con 0x7f616c06ab80 2024-03-25T19:24:12.169 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:12.169+0000 7f6153fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:12.170 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:12.169+0000 7f6153fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:24:12.394 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:12.394+0000 7f6171ced640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:12.394 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:12.394+0000 7f6171ced640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:12.394 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:12.394+0000 7f6171ced640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:24:12.394 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:12.394+0000 7f6171ced640 1 -- 172.21.15.83:0/3653634678 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f616c0627b0 con 0x7f616c06ab80 2024-03-25T19:24:12.431 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:12.431+0000 7f6153fff640 1 -- 172.21.15.83:0/3653634678 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7f615c052280 con 0x7f616c06ab80 2024-03-25T19:24:12.431 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:12.432+0000 7f6153fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:12.431 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:12.432+0000 7f6153fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:24:12.432 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:24:12.432 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:24:12.435 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:12.434+0000 7f6171ced640 10 librados: watch_flush enter 2024-03-25T19:24:12.435 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:12.434+0000 7f6171ced640 10 librados: watch_flush exit 2024-03-25T19:24:12.435 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:12.434+0000 7f6171ced640 20 client.4422.objecter shutdown clearing up homeless session... 2024-03-25T19:24:12.435 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:12.434+0000 7f6171ced640 10 client.4422.objecter successfully canceled tick 2024-03-25T19:24:12.435 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:12.434+0000 7f6171ced640 1 -- 172.21.15.83:0/3653634678 >> v1:172.21.15.103:6800/345072327 conn(0x7f615806e160 legacy=0x7f6158070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:12.435 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:12.434+0000 7f6171ced640 10 monclient: shutdown 2024-03-25T19:24:12.435 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:12.434+0000 7f6171ced640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:12.435 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:12.434+0000 7f6171ced640 1 -- 172.21.15.83:0/3653634678 >> v1:172.21.15.83:6789/0 conn(0x7f616c06ab80 legacy=0x7f616c1ce900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:12.435 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:12.434+0000 7f6171ced640 1 -- 172.21.15.83:0/3653634678 shutdown_connections 2024-03-25T19:24:12.435 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:12.434+0000 7f6171ced640 1 -- 172.21.15.83:0/3653634678 >> 172.21.15.83:0/3653634678 conn(0x7f616c0b4a80 msgr2=0x7f616c0adfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:12.435 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:12.434+0000 7f6171ced640 1 -- 172.21.15.83:0/3653634678 shutdown_connections 2024-03-25T19:24:12.435 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:12.434+0000 7f6171ced640 1 -- 172.21.15.83:0/3653634678 wait complete. 2024-03-25T19:24:12.435 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:12.434+0000 7f6171ced640 1 librados: shutdown 2024-03-25T19:24:12.445 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:24:19.253 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:24:19.362 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.363+0000 7f14d67f4640 10 monclient: get_monmap_and_config 2024-03-25T19:24:19.362 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.363+0000 7f14d67f4640 10 monclient: build_initial_monmap 2024-03-25T19:24:19.363 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.363+0000 7f14d67f4640 10 monclient: monmap: 2024-03-25T19:24:19.363 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:24:19.363 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:24:19.363 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:24:19.364048+0000 2024-03-25T19:24:19.363 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:24:19.364048+0000 2024-03-25T19:24:19.363 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:19.363 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:24:19.363 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:24:19.363 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:24:19.363 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:24:19.363 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:19.364 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.364+0000 7f14d67f4640 1 Processor -- start 2024-03-25T19:24:19.364 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.364+0000 7f14d67f4640 1 -- start start 2024-03-25T19:24:19.364 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.364+0000 7f14d67f4640 10 monclient: init 2024-03-25T19:24:19.364 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.365+0000 7f14d67f4640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:19.365 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.365+0000 7f14d67f4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:24:19.365 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.365+0000 7f14d67f4640 10 monclient(hunting): picked mon.noname-c con 0x7f14d00a98f0 addr v1:172.21.15.103:6789/0 2024-03-25T19:24:19.365 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.365+0000 7f14d67f4640 10 monclient(hunting): picked mon.noname-b con 0x7f14d013f710 addr v1:172.21.15.83:6790/0 2024-03-25T19:24:19.365 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.365+0000 7f14d67f4640 10 monclient(hunting): picked mon.noname-a con 0x7f14d01430c0 addr v1:172.21.15.83:6789/0 2024-03-25T19:24:19.365 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.365+0000 7f14d67f4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14d0142520 con 0x7f14d01430c0 2024-03-25T19:24:19.365 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.365+0000 7f14d67f4640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14d00b28a0 con 0x7f14d00a98f0 2024-03-25T19:24:19.365 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.365+0000 7f14d67f4640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14d00b2a30 con 0x7f14d013f710 2024-03-25T19:24:19.365 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.365+0000 7f14d67f4640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:19.365 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.365+0000 7f14d67f4640 10 monclient(hunting): authenticate will time out at 6316.636872s 2024-03-25T19:24:19.372 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.373+0000 7f14cffff640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f14d00a98f0 0x7f14d00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:59560/0 (socket says 172.21.15.83:59560) 2024-03-25T19:24:19.372 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.373+0000 7f14cffff640 1 -- 172.21.15.83:0/97433031 learned_addr learned my addr 172.21.15.83:0/97433031 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:24:19.389 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.389+0000 7f14ceffd640 1 -- 172.21.15.83:0/97433031 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2064968769 0 0) 0x7f14d0142520 con 0x7f14d01430c0 2024-03-25T19:24:19.389 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.389+0000 7f14ceffd640 10 monclient(hunting): _negotiate 2024-03-25T19:24:19.389 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.389+0000 7f14ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:19.389 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.389+0000 7f14ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:19.389 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.389+0000 7f14ceffd640 10 monclient(hunting): my global_id is 4425 2024-03-25T19:24:19.389 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.389+0000 7f14ceffd640 1 -- 172.21.15.83:0/97433031 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14b8003520 con 0x7f14d01430c0 2024-03-25T19:24:19.392 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.392+0000 7f14ceffd640 1 -- 172.21.15.83:0/97433031 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3323171337 0 0) 0x7f14d00b28a0 con 0x7f14d00a98f0 2024-03-25T19:24:19.392 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.392+0000 7f14ceffd640 10 monclient(hunting): _negotiate 2024-03-25T19:24:19.393 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.392+0000 7f14ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:19.393 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.392+0000 7f14ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:19.393 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.392+0000 7f14ceffd640 10 monclient(hunting): my global_id is 4432 2024-03-25T19:24:19.393 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.392+0000 7f14ceffd640 1 -- 172.21.15.83:0/97433031 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14d0142520 con 0x7f14d00a98f0 2024-03-25T19:24:19.393 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.393+0000 7f14ceffd640 1 -- 172.21.15.83:0/97433031 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2823835857 0 0) 0x7f14d00b2a30 con 0x7f14d013f710 2024-03-25T19:24:19.393 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.393+0000 7f14ceffd640 10 monclient(hunting): _negotiate 2024-03-25T19:24:19.393 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.393+0000 7f14ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:19.393 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.393+0000 7f14ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:19.393 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.393+0000 7f14ceffd640 10 monclient(hunting): my global_id is 4439 2024-03-25T19:24:19.393 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.393+0000 7f14ceffd640 1 -- 172.21.15.83:0/97433031 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14d00b28a0 con 0x7f14d013f710 2024-03-25T19:24:19.410 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.411+0000 7f14ceffd640 1 -- 172.21.15.83:0/97433031 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4015931642 0 0) 0x7f14d00b28a0 con 0x7f14d013f710 2024-03-25T19:24:19.411 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.411+0000 7f14ceffd640 1 -- 172.21.15.83:0/97433031 >> v1:172.21.15.103:6789/0 conn(0x7f14d00a98f0 legacy=0x7f14d00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:19.411 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.411+0000 7f14ceffd640 1 -- 172.21.15.83:0/97433031 >> v1:172.21.15.83:6789/0 conn(0x7f14d01430c0 legacy=0x7f14d014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:19.411 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.411+0000 7f14ceffd640 10 monclient: _finish_hunting 0 2024-03-25T19:24:19.411 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.411+0000 7f14ceffd640 1 monclient: found mon.noname-b 2024-03-25T19:24:19.411 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.411+0000 7f14ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:19.411 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.411+0000 7f14ceffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.83:6790/0 2024-03-25T19:24:19.411 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.411+0000 7f14ceffd640 1 -- 172.21.15.83:0/97433031 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d00b44c0 con 0x7f14d013f710 2024-03-25T19:24:19.411 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.411+0000 7f14ceffd640 10 monclient: _finish_auth 0 2024-03-25T19:24:19.411 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.411+0000 7f14ceffd640 10 monclient: _check_auth_tickets 2024-03-25T19:24:19.411 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.411+0000 7f14ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:19.411 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.411+0000 7f14ceffd640 1 -- 172.21.15.83:0/97433031 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f14c40015b0 con 0x7f14d013f710 2024-03-25T19:24:19.411 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.411+0000 7f14ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:19.411 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.411+0000 7f14ceffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:24:19.411 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.411+0000 7f14ceffd640 10 monclient: dump: 2024-03-25T19:24:19.411 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:24:19.411 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:24:19.412 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:19.412 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:19.412 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:19.412 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:24:19.412 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:24:19.412 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:24:19.412 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:24:19.412 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:19.412 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.411+0000 7f14d67f4640 5 monclient: authenticate success, global_id 4439 2024-03-25T19:24:19.412 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.411+0000 7f14d67f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:24:19.419 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.419+0000 7f14ceffd640 1 -- 172.21.15.83:0/97433031 <== mon.2 v1:172.21.15.83:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f14c4001790 con 0x7f14d013f710 2024-03-25T19:24:19.419 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.419+0000 7f14ceffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:19.419 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.419+0000 7f14ceffd640 1 -- 172.21.15.83:0/97433031 <== mon.2 v1:172.21.15.83:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f14c4001ba0 con 0x7f14d013f710 2024-03-25T19:24:19.419 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.419+0000 7f14ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:19.419 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.419+0000 7f14ceffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:24:19.419 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.419+0000 7f14ceffd640 10 monclient: dump: 2024-03-25T19:24:19.419 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:24:19.419 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:24:19.420 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:19.420 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:19.420 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:19.420 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:24:19.420 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:24:19.420 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:24:19.420 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:24:19.420 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:19.420 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.419+0000 7f14d67f4640 10 monclient: get_monmap_and_config success 2024-03-25T19:24:19.420 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.419+0000 7f14d67f4640 4 set_mon_vals no callback set 2024-03-25T19:24:19.421 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.420+0000 7f14d67f4640 10 monclient: shutdown 2024-03-25T19:24:19.421 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.420+0000 7f14d67f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:19.421 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.420+0000 7f14d67f4640 1 -- 172.21.15.83:0/97433031 >> v1:172.21.15.83:6790/0 conn(0x7f14d013f710 legacy=0x7f14d01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:19.421 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.420+0000 7f14d67f4640 1 -- 172.21.15.83:0/97433031 shutdown_connections 2024-03-25T19:24:19.421 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.420+0000 7f14d67f4640 1 -- 172.21.15.83:0/97433031 >> 172.21.15.83:0/97433031 conn(0x7f14d00b7ea0 msgr2=0x7f14d00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:19.421 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.420+0000 7f14d67f4640 1 -- 172.21.15.83:0/97433031 shutdown_connections 2024-03-25T19:24:19.421 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.420+0000 7f14d67f4640 1 -- 172.21.15.83:0/97433031 wait complete. 2024-03-25T19:24:19.422 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.421+0000 7f14d67f4640 10 monclient: build_initial_monmap 2024-03-25T19:24:19.422 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.421+0000 7f14d67f4640 10 monclient: monmap: 2024-03-25T19:24:19.422 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:24:19.422 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:24:19.422 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:24:19.422 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:24:19.422 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:19.422 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:24:19.422 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:24:19.422 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:24:19.422 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:24:19.422 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:19.423 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.422+0000 7f14d67f4640 1 librados: starting msgr at 2024-03-25T19:24:19.424 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.422+0000 7f14d67f4640 1 librados: starting objecter 2024-03-25T19:24:19.424 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.422+0000 7f14d67f4640 1 Processor -- start 2024-03-25T19:24:19.424 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.422+0000 7f14d67f4640 1 -- start start 2024-03-25T19:24:19.424 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.422+0000 7f14d67f4640 1 librados: setting wanted keys 2024-03-25T19:24:19.424 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.422+0000 7f14d67f4640 1 librados: calling monclient init 2024-03-25T19:24:19.424 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.422+0000 7f14d67f4640 10 monclient: init 2024-03-25T19:24:19.424 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.423+0000 7f14d67f4640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:19.424 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.423+0000 7f14d67f4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:24:19.424 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.423+0000 7f14d67f4640 10 monclient(hunting): picked mon.noname-b con 0x7f14d00a98f0 addr v1:172.21.15.103:6789/0 2024-03-25T19:24:19.424 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.423+0000 7f14d67f4640 10 monclient(hunting): picked mon.noname-a con 0x7f14d013f710 addr v1:172.21.15.83:6789/0 2024-03-25T19:24:19.424 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.423+0000 7f14d67f4640 10 monclient(hunting): picked mon.noname-c con 0x7f14d01430c0 addr v1:172.21.15.83:6790/0 2024-03-25T19:24:19.424 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.423+0000 7f14d67f4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14d01349e0 con 0x7f14d013f710 2024-03-25T19:24:19.424 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.423+0000 7f14d67f4640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14d01397c0 con 0x7f14d00a98f0 2024-03-25T19:24:19.424 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.423+0000 7f14d67f4640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14d0134e70 con 0x7f14d01430c0 2024-03-25T19:24:19.424 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.423+0000 7f14d67f4640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:19.424 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.423+0000 7f14d67f4640 10 monclient(hunting): authenticate will time out at 6316.695038s 2024-03-25T19:24:19.432 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.433+0000 7f14cffff640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f14d00a98f0 0x7f14d01d7a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:59616/0 (socket says 172.21.15.83:59616) 2024-03-25T19:24:19.432 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.433+0000 7f14cffff640 1 -- 172.21.15.83:0/643864819 learned_addr learned my addr 172.21.15.83:0/643864819 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:24:19.434 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.435+0000 7f14ccff9640 10 client.?.objecter ms_handle_connect 0x7f14d01430c0 2024-03-25T19:24:19.434 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.435+0000 7f14ccff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:19.435 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.435+0000 7f14ccff9640 10 client.?.objecter ms_handle_connect 0x7f14d00a98f0 2024-03-25T19:24:19.435 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.435+0000 7f14ccff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:19.439 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.440+0000 7f14ccff9640 10 client.?.objecter ms_handle_connect 0x7f14d013f710 2024-03-25T19:24:19.439 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.440+0000 7f14ccff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:19.443 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.443+0000 7f14ccff9640 1 -- 172.21.15.83:0/643864819 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2337770884 0 0) 0x7f14d0134e70 con 0x7f14d01430c0 2024-03-25T19:24:19.443 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.443+0000 7f14ccff9640 10 monclient(hunting): _negotiate 2024-03-25T19:24:19.443 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.443+0000 7f14ccff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:19.443 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.443+0000 7f14ccff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:19.443 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.443+0000 7f14ccff9640 10 monclient(hunting): my global_id is 4442 2024-03-25T19:24:19.443 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.443+0000 7f14ccff9640 1 -- 172.21.15.83:0/643864819 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14bc004eb0 con 0x7f14d01430c0 2024-03-25T19:24:19.446 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.447+0000 7f14ccff9640 1 -- 172.21.15.83:0/643864819 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3873292892 0 0) 0x7f14d01397c0 con 0x7f14d00a98f0 2024-03-25T19:24:19.447 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.447+0000 7f14ccff9640 10 monclient(hunting): _negotiate 2024-03-25T19:24:19.447 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.447+0000 7f14ccff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:19.447 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.447+0000 7f14ccff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:19.447 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.447+0000 7f14ccff9640 10 monclient(hunting): my global_id is 4435 2024-03-25T19:24:19.447 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.447+0000 7f14ccff9640 1 -- 172.21.15.83:0/643864819 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14d0134e70 con 0x7f14d00a98f0 2024-03-25T19:24:19.452 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.452+0000 7f14ccff9640 1 -- 172.21.15.83:0/643864819 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1616474570 0 0) 0x7f14d01349e0 con 0x7f14d013f710 2024-03-25T19:24:19.452 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.452+0000 7f14ccff9640 10 monclient(hunting): _negotiate 2024-03-25T19:24:19.452 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.452+0000 7f14ccff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:19.452 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.452+0000 7f14ccff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:19.452 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.452+0000 7f14ccff9640 10 monclient(hunting): my global_id is 4428 2024-03-25T19:24:19.452 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.452+0000 7f14ccff9640 1 -- 172.21.15.83:0/643864819 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14d01397c0 con 0x7f14d013f710 2024-03-25T19:24:19.477 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.477+0000 7f14ccff9640 1 -- 172.21.15.83:0/643864819 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2793572619 0 0) 0x7f14bc004eb0 con 0x7f14d01430c0 2024-03-25T19:24:19.477 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.477+0000 7f14ccff9640 1 -- 172.21.15.83:0/643864819 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14d01349e0 con 0x7f14d01430c0 2024-03-25T19:24:19.477 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.477+0000 7f14ccff9640 1 -- 172.21.15.83:0/643864819 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f14c8001b30 con 0x7f14d01430c0 2024-03-25T19:24:19.477 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.477+0000 7f14ccff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:19.477 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.477+0000 7f14ccff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:24:19.477 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.477+0000 7f14ccff9640 10 monclient(hunting): dump: 2024-03-25T19:24:19.477 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:24:19.477 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:24:19.477 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:19.477 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:19.477 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:19.478 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:24:19.478 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:24:19.478 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:24:19.478 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:24:19.478 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:19.478 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.477+0000 7f14ccff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:24:19.478 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.478+0000 7f14ccff9640 1 -- 172.21.15.83:0/643864819 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1525666128 0 0) 0x7f14d0134e70 con 0x7f14d00a98f0 2024-03-25T19:24:19.478 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.478+0000 7f14ccff9640 1 -- 172.21.15.83:0/643864819 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14bc004eb0 con 0x7f14d00a98f0 2024-03-25T19:24:19.478 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.478+0000 7f14ccff9640 1 -- 172.21.15.83:0/643864819 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f14c0001930 con 0x7f14d00a98f0 2024-03-25T19:24:19.478 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.478+0000 7f14ccff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:19.479 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.478+0000 7f14ccff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:24:19.479 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.478+0000 7f14ccff9640 10 monclient(hunting): dump: 2024-03-25T19:24:19.479 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:24:19.479 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:24:19.479 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:19.479 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:19.479 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:19.479 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:24:19.479 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:24:19.479 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:24:19.479 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:24:19.479 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:19.485 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.485+0000 7f14ccff9640 1 -- 172.21.15.83:0/643864819 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2546377609 0 0) 0x7f14d01397c0 con 0x7f14d013f710 2024-03-25T19:24:19.485 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.485+0000 7f14ccff9640 1 -- 172.21.15.83:0/643864819 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14d0134e70 con 0x7f14d013f710 2024-03-25T19:24:19.485 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.485+0000 7f14ccff9640 1 -- 172.21.15.83:0/643864819 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f14c4001aa0 con 0x7f14d013f710 2024-03-25T19:24:19.485 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.485+0000 7f14ccff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:19.485 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.485+0000 7f14ccff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:24:19.485 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.485+0000 7f14ccff9640 10 monclient(hunting): dump: 2024-03-25T19:24:19.485 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:24:19.485 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:24:19.485 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:19.485 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:19.485 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:19.485 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:24:19.485 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:24:19.486 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:24:19.486 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:24:19.486 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:19.495 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.495+0000 7f14ccff9640 1 -- 172.21.15.83:0/643864819 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2692653735 0 0) 0x7f14d01349e0 con 0x7f14d01430c0 2024-03-25T19:24:19.495 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.495+0000 7f14ccff9640 1 -- 172.21.15.83:0/643864819 >> v1:172.21.15.103:6789/0 conn(0x7f14d00a98f0 legacy=0x7f14d01d7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:19.495 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.495+0000 7f14ccff9640 1 -- 172.21.15.83:0/643864819 >> v1:172.21.15.83:6789/0 conn(0x7f14d013f710 legacy=0x7f14d0133bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:19.495 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.495+0000 7f14ccff9640 10 monclient: _finish_hunting 0 2024-03-25T19:24:19.495 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.495+0000 7f14ccff9640 1 monclient: found mon.c 2024-03-25T19:24:19.496 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.495+0000 7f14ccff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:19.496 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.495+0000 7f14ccff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:24:19.496 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.495+0000 7f14ccff9640 1 -- 172.21.15.83:0/643864819 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d0135000 con 0x7f14d01430c0 2024-03-25T19:24:19.496 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.495+0000 7f14ccff9640 10 monclient: _finish_auth 0 2024-03-25T19:24:19.496 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.495+0000 7f14ccff9640 10 monclient: _check_auth_tickets 2024-03-25T19:24:19.496 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.495+0000 7f14ccff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:19.496 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.495+0000 7f14d67f4640 5 monclient: authenticate success, global_id 4442 2024-03-25T19:24:19.496 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.495+0000 7f14d67f4640 10 monclient: _renew_subs 2024-03-25T19:24:19.496 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.495+0000 7f14d67f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:24:19.496 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.495+0000 7f14d67f4640 1 -- 172.21.15.83:0/643864819 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14d01d8140 con 0x7f14d01430c0 2024-03-25T19:24:19.496 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.495+0000 7f14d67f4640 10 client.4442.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:24:19.496 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.495+0000 7f14d67f4640 10 monclient: _renew_subs 2024-03-25T19:24:19.496 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.495+0000 7f14d67f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:24:19.496 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.495+0000 7f14d67f4640 1 -- 172.21.15.83:0/643864819 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f14d0137900 con 0x7f14d01430c0 2024-03-25T19:24:19.496 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.495+0000 7f14d67f4640 1 librados: init done 2024-03-25T19:24:19.497 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.496+0000 7f14d67f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:19.497 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.496+0000 7f14d67f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:19.497 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.496+0000 7f14d67f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:24:19.497 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.496+0000 7f14d67f4640 1 -- 172.21.15.83:0/643864819 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14d0074cc0 con 0x7f14d01430c0 2024-03-25T19:24:19.807 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.808+0000 7f14ccff9640 1 -- 172.21.15.83:0/643864819 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f14c8002000 con 0x7f14d01430c0 2024-03-25T19:24:19.807 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.808+0000 7f14ccff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:19.808 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.808+0000 7f14ccff9640 1 -- 172.21.15.83:0/643864819 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f14c8003b10 con 0x7f14d01430c0 2024-03-25T19:24:19.808 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.808+0000 7f14ccff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:19.808 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.808+0000 7f14ccff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:24:19.808 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.808+0000 7f14ccff9640 10 monclient: dump: 2024-03-25T19:24:19.808 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:24:19.808 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:24:19.808 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:19.808 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:19.808 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:19.808 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:24:19.808 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:24:19.808 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:24:19.808 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:24:19.808 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:19.808 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.808+0000 7f14ce7fc640 4 set_mon_vals no callback set 2024-03-25T19:24:19.809 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.808+0000 7f14ccff9640 1 -- 172.21.15.83:0/643864819 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7f14c8019ae0 con 0x7f14d01430c0 2024-03-25T19:24:19.809 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.809+0000 7f14cffff640 10 monclient: get_auth_request con 0x7f14bc06e0f0 auth_method 0 2024-03-25T19:24:19.809 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.809+0000 7f14ccff9640 10 client.4442.objecter ms_handle_connect 0x7f14bc06e0f0 2024-03-25T19:24:19.814 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.814+0000 7f14ccff9640 1 -- 172.21.15.83:0/643864819 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7f14c8053730 con 0x7f14d01430c0 2024-03-25T19:24:19.815 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.814+0000 7f14ccff9640 10 client.4442.objecter ms_dispatch 0x7f14d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:24:19.815 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.814+0000 7f14ccff9640 3 client.4442.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:24:19.815 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.814+0000 7f14ccff9640 3 client.4442.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:24:19.815 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.815+0000 7f14ccff9640 20 client.4442.objecter dump_active .. 0 homeless 2024-03-25T19:24:19.815 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.815+0000 7f14ccff9640 1 -- 172.21.15.83:0/643864819 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f14c804f240 con 0x7f14d01430c0 2024-03-25T19:24:19.815 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.815+0000 7f14ccff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:19.815 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:19.815+0000 7f14ccff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:24:20.081 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:20.082+0000 7f14d67f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:20.082 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:20.082+0000 7f14d67f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:20.082 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:20.082+0000 7f14d67f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:24:20.082 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:20.082+0000 7f14d67f4640 1 -- 172.21.15.83:0/643864819 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f14d00627b0 con 0x7f14d01430c0 2024-03-25T19:24:20.124 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:20.124+0000 7f14ccff9640 1 -- 172.21.15.83:0/643864819 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7f14c8051170 con 0x7f14d01430c0 2024-03-25T19:24:20.124 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:20.124+0000 7f14ccff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:20.124 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:20.124+0000 7f14ccff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:24:20.124 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:24:20.124 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:24:20.127 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:20.127+0000 7f14d67f4640 10 librados: watch_flush enter 2024-03-25T19:24:20.127 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:20.127+0000 7f14d67f4640 10 librados: watch_flush exit 2024-03-25T19:24:20.127 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:20.127+0000 7f14d67f4640 20 client.4442.objecter shutdown clearing up homeless session... 2024-03-25T19:24:20.127 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:20.127+0000 7f14d67f4640 10 client.4442.objecter successfully canceled tick 2024-03-25T19:24:20.127 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:20.127+0000 7f14d67f4640 1 -- 172.21.15.83:0/643864819 >> v1:172.21.15.103:6800/345072327 conn(0x7f14bc06e0f0 legacy=0x7f14bc0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:20.127 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:20.127+0000 7f14d67f4640 10 monclient: shutdown 2024-03-25T19:24:20.127 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:20.127+0000 7f14d67f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:20.127 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:20.127+0000 7f14d67f4640 1 -- 172.21.15.83:0/643864819 >> v1:172.21.15.83:6790/0 conn(0x7f14d01430c0 legacy=0x7f14d01342d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:20.128 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:20.127+0000 7f14d67f4640 1 -- 172.21.15.83:0/643864819 shutdown_connections 2024-03-25T19:24:20.128 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:20.127+0000 7f14d67f4640 1 -- 172.21.15.83:0/643864819 >> 172.21.15.83:0/643864819 conn(0x7f14d00b7ea0 msgr2=0x7f14d0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:20.128 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:20.127+0000 7f14d67f4640 1 -- 172.21.15.83:0/643864819 shutdown_connections 2024-03-25T19:24:20.128 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:20.127+0000 7f14d67f4640 1 -- 172.21.15.83:0/643864819 wait complete. 2024-03-25T19:24:20.128 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:20.127+0000 7f14d67f4640 1 librados: shutdown 2024-03-25T19:24:20.138 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:24:22.932 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:05:04.859 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:22.982 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:05:04.910 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:23.023 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:05:04.951 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:23.028 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:05:04.938 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:23.033 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:05:04.943 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:23.057 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:05:04.968 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:23.072 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:05:04.999 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:23.076 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:05:05.004 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:23.098 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:05:05.025 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:26.942 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:24:27.055 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.055+0000 7fb6a3ff4640 10 monclient: get_monmap_and_config 2024-03-25T19:24:27.055 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.056+0000 7fb6a3ff4640 10 monclient: build_initial_monmap 2024-03-25T19:24:27.055 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.056+0000 7fb6a3ff4640 10 monclient: monmap: 2024-03-25T19:24:27.055 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:24:27.055 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:24:27.056 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:24:27.056941+0000 2024-03-25T19:24:27.056 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:24:27.056941+0000 2024-03-25T19:24:27.056 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:27.056 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:24:27.056 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:24:27.056 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:24:27.056 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:24:27.056 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:27.057 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.056+0000 7fb6a3ff4640 1 Processor -- start 2024-03-25T19:24:27.057 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.056+0000 7fb6a3ff4640 1 -- start start 2024-03-25T19:24:27.057 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.056+0000 7fb6a3ff4640 10 monclient: init 2024-03-25T19:24:27.057 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.057+0000 7fb6a3ff4640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:27.057 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.057+0000 7fb6a3ff4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:24:27.057 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.057+0000 7fb6a3ff4640 10 monclient(hunting): picked mon.noname-c con 0x7fb69c06adf0 addr v1:172.21.15.103:6789/0 2024-03-25T19:24:27.057 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.057+0000 7fb6a3ff4640 10 monclient(hunting): picked mon.noname-b con 0x7fb69c0af580 addr v1:172.21.15.83:6790/0 2024-03-25T19:24:27.057 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.057+0000 7fb6a3ff4640 10 monclient(hunting): picked mon.noname-a con 0x7fb69c1409d0 addr v1:172.21.15.83:6789/0 2024-03-25T19:24:27.057 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.057+0000 7fb6a3ff4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb69c0b0b20 con 0x7fb69c1409d0 2024-03-25T19:24:27.057 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.057+0000 7fb6a3ff4640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb69c0afce0 con 0x7fb69c06adf0 2024-03-25T19:24:27.058 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.057+0000 7fb6a3ff4640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb69c0af9e0 con 0x7fb69c0af580 2024-03-25T19:24:27.058 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.057+0000 7fb6a3ff4640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:27.058 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.057+0000 7fb6a3ff4640 10 monclient(hunting): authenticate will time out at 6324.329307s 2024-03-25T19:24:27.061 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.061+0000 7fb6a158f640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fb69c0af580 0x7fb69c1402c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:50880/0 (socket says 172.21.15.83:50880) 2024-03-25T19:24:27.061 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.061+0000 7fb6a158f640 1 -- 172.21.15.83:0/138530730 learned_addr learned my addr 172.21.15.83:0/138530730 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:24:27.072 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.072+0000 7fb6a0d8e640 1 -- 172.21.15.83:0/138530730 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4249767122 0 0) 0x7fb69c0b0b20 con 0x7fb69c1409d0 2024-03-25T19:24:27.072 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.072+0000 7fb6a0d8e640 10 monclient(hunting): _negotiate 2024-03-25T19:24:27.072 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.072+0000 7fb6a0d8e640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:27.072 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.072+0000 7fb6a0d8e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:27.072 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.072+0000 7fb6a0d8e640 10 monclient(hunting): my global_id is 4431 2024-03-25T19:24:27.072 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.072+0000 7fb6a0d8e640 1 -- 172.21.15.83:0/138530730 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb688003520 con 0x7fb69c1409d0 2024-03-25T19:24:27.080 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.081+0000 7fb6a0d8e640 1 -- 172.21.15.83:0/138530730 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3368416049 0 0) 0x7fb69c0af9e0 con 0x7fb69c0af580 2024-03-25T19:24:27.080 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.081+0000 7fb6a0d8e640 10 monclient(hunting): _negotiate 2024-03-25T19:24:27.081 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.081+0000 7fb6a0d8e640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:27.081 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.081+0000 7fb6a0d8e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:27.081 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.081+0000 7fb6a0d8e640 10 monclient(hunting): my global_id is 4445 2024-03-25T19:24:27.081 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.081+0000 7fb6a0d8e640 1 -- 172.21.15.83:0/138530730 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb69c0b0b20 con 0x7fb69c0af580 2024-03-25T19:24:27.090 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.091+0000 7fb6a0d8e640 1 -- 172.21.15.83:0/138530730 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2879296751 0 0) 0x7fb69c0afce0 con 0x7fb69c06adf0 2024-03-25T19:24:27.090 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.091+0000 7fb6a0d8e640 10 monclient(hunting): _negotiate 2024-03-25T19:24:27.091 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.091+0000 7fb6a0d8e640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:27.091 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.091+0000 7fb6a0d8e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:27.091 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.091+0000 7fb6a0d8e640 10 monclient(hunting): my global_id is 4438 2024-03-25T19:24:27.091 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.091+0000 7fb6a0d8e640 1 -- 172.21.15.83:0/138530730 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb69c0af9e0 con 0x7fb69c06adf0 2024-03-25T19:24:27.101 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.101+0000 7fb6a0d8e640 1 -- 172.21.15.83:0/138530730 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2732246462 0 0) 0x7fb69c0b0b20 con 0x7fb69c0af580 2024-03-25T19:24:27.101 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.101+0000 7fb6a0d8e640 1 -- 172.21.15.83:0/138530730 >> v1:172.21.15.103:6789/0 conn(0x7fb69c06adf0 legacy=0x7fb69c13d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:27.101 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.101+0000 7fb6a0d8e640 1 -- 172.21.15.83:0/138530730 >> v1:172.21.15.83:6789/0 conn(0x7fb69c1409d0 legacy=0x7fb69c14b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:27.101 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.101+0000 7fb6a0d8e640 10 monclient: _finish_hunting 0 2024-03-25T19:24:27.102 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.101+0000 7fb6a0d8e640 1 monclient: found mon.noname-b 2024-03-25T19:24:27.102 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.101+0000 7fb6a0d8e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:27.102 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.101+0000 7fb6a0d8e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.83:6790/0 2024-03-25T19:24:27.102 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.101+0000 7fb6a0d8e640 1 -- 172.21.15.83:0/138530730 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb69c13dd40 con 0x7fb69c0af580 2024-03-25T19:24:27.102 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.101+0000 7fb6a0d8e640 10 monclient: _finish_auth 0 2024-03-25T19:24:27.102 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.101+0000 7fb6a0d8e640 10 monclient: _check_auth_tickets 2024-03-25T19:24:27.102 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.101+0000 7fb6a0d8e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:27.102 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.101+0000 7fb6a0d8e640 1 -- 172.21.15.83:0/138530730 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fb68c001670 con 0x7fb69c0af580 2024-03-25T19:24:27.102 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.101+0000 7fb6a3ff4640 5 monclient: authenticate success, global_id 4445 2024-03-25T19:24:27.102 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.101+0000 7fb6a0d8e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:27.102 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.101+0000 7fb6a0d8e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:24:27.102 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.101+0000 7fb6a0d8e640 10 monclient: dump: 2024-03-25T19:24:27.102 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:24:27.103 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:24:27.103 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:27.103 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:27.103 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:27.103 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:24:27.103 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:24:27.103 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:24:27.103 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:24:27.103 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:27.103 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.101+0000 7fb6a3ff4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:24:27.108 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.109+0000 7fb6a0d8e640 1 -- 172.21.15.83:0/138530730 <== mon.2 v1:172.21.15.83:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb68c001890 con 0x7fb69c0af580 2024-03-25T19:24:27.108 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.109+0000 7fb6a0d8e640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:27.108 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.109+0000 7fb6a0d8e640 1 -- 172.21.15.83:0/138530730 <== mon.2 v1:172.21.15.83:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fb68c001ba0 con 0x7fb69c0af580 2024-03-25T19:24:27.108 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.109+0000 7fb6a3ff4640 10 monclient: get_monmap_and_config success 2024-03-25T19:24:27.109 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.109+0000 7fb6a3ff4640 4 set_mon_vals no callback set 2024-03-25T19:24:27.109 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.109+0000 7fb6a0d8e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:27.109 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.109+0000 7fb6a0d8e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:24:27.109 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.109+0000 7fb6a0d8e640 10 monclient: dump: 2024-03-25T19:24:27.109 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:24:27.109 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:24:27.109 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:27.109 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:27.109 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:27.109 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:24:27.109 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:24:27.109 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:24:27.109 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:24:27.109 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:27.111 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.109+0000 7fb6a3ff4640 10 monclient: shutdown 2024-03-25T19:24:27.111 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.109+0000 7fb6a3ff4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:27.111 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.109+0000 7fb6a3ff4640 1 -- 172.21.15.83:0/138530730 >> v1:172.21.15.83:6790/0 conn(0x7fb69c0af580 legacy=0x7fb69c1402c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:27.111 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.110+0000 7fb6a3ff4640 1 -- 172.21.15.83:0/138530730 shutdown_connections 2024-03-25T19:24:27.111 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.110+0000 7fb6a3ff4640 1 -- 172.21.15.83:0/138530730 >> 172.21.15.83:0/138530730 conn(0x7fb69c0b4b80 msgr2=0x7fb69c0b4f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:27.111 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.110+0000 7fb6a3ff4640 1 -- 172.21.15.83:0/138530730 shutdown_connections 2024-03-25T19:24:27.111 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.110+0000 7fb6a3ff4640 1 -- 172.21.15.83:0/138530730 wait complete. 2024-03-25T19:24:27.111 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.110+0000 7fb6a3ff4640 10 monclient: build_initial_monmap 2024-03-25T19:24:27.111 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.110+0000 7fb6a3ff4640 10 monclient: monmap: 2024-03-25T19:24:27.111 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:24:27.111 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:24:27.111 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:24:27.111 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:24:27.112 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:27.112 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:24:27.112 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:24:27.112 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:24:27.112 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:24:27.112 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:27.112 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.111+0000 7fb6a3ff4640 1 librados: starting msgr at 2024-03-25T19:24:27.112 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.111+0000 7fb6a3ff4640 1 librados: starting objecter 2024-03-25T19:24:27.112 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.111+0000 7fb6a3ff4640 1 Processor -- start 2024-03-25T19:24:27.112 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.111+0000 7fb6a3ff4640 1 -- start start 2024-03-25T19:24:27.112 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.111+0000 7fb6a3ff4640 1 librados: setting wanted keys 2024-03-25T19:24:27.112 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.111+0000 7fb6a3ff4640 1 librados: calling monclient init 2024-03-25T19:24:27.112 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.111+0000 7fb6a3ff4640 10 monclient: init 2024-03-25T19:24:27.113 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.112+0000 7fb6a3ff4640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:27.113 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.112+0000 7fb6a3ff4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:24:27.113 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.112+0000 7fb6a3ff4640 10 monclient(hunting): picked mon.noname-b con 0x7fb69c06adf0 addr v1:172.21.15.103:6789/0 2024-03-25T19:24:27.113 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.112+0000 7fb6a3ff4640 10 monclient(hunting): picked mon.noname-c con 0x7fb69c0af580 addr v1:172.21.15.83:6790/0 2024-03-25T19:24:27.114 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.112+0000 7fb6a3ff4640 10 monclient(hunting): picked mon.noname-a con 0x7fb69c1409d0 addr v1:172.21.15.83:6789/0 2024-03-25T19:24:27.114 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.112+0000 7fb6a3ff4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb69c142e30 con 0x7fb69c1409d0 2024-03-25T19:24:27.114 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.112+0000 7fb6a3ff4640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb69c1438b0 con 0x7fb69c06adf0 2024-03-25T19:24:27.114 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.112+0000 7fb6a3ff4640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb69c143d30 con 0x7fb69c0af580 2024-03-25T19:24:27.114 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.112+0000 7fb6a3ff4640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:27.114 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.112+0000 7fb6a3ff4640 10 monclient(hunting): authenticate will time out at 6324.384311s 2024-03-25T19:24:27.115 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.115+0000 7fb6a1d90640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fb69c06adf0 0x7fb69c1430c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:59350/0 (socket says 172.21.15.83:59350) 2024-03-25T19:24:27.115 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.115+0000 7fb6a1d90640 1 -- 172.21.15.83:0/2356941783 learned_addr learned my addr 172.21.15.83:0/2356941783 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:24:27.117 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.118+0000 7fb6927fc640 10 client.?.objecter ms_handle_connect 0x7fb69c06adf0 2024-03-25T19:24:27.117 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.118+0000 7fb6927fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:27.121 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.122+0000 7fb6927fc640 10 client.?.objecter ms_handle_connect 0x7fb69c1409d0 2024-03-25T19:24:27.121 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.122+0000 7fb6927fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:27.131 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.131+0000 7fb6927fc640 10 client.?.objecter ms_handle_connect 0x7fb69c0af580 2024-03-25T19:24:27.131 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.131+0000 7fb6927fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:27.133 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.133+0000 7fb6927fc640 1 -- 172.21.15.83:0/2356941783 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3462761448 0 0) 0x7fb69c142e30 con 0x7fb69c1409d0 2024-03-25T19:24:27.133 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.133+0000 7fb6927fc640 10 monclient(hunting): _negotiate 2024-03-25T19:24:27.133 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.133+0000 7fb6927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:27.133 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.133+0000 7fb6927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:27.134 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.133+0000 7fb6927fc640 10 monclient(hunting): my global_id is 4434 2024-03-25T19:24:27.134 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.134+0000 7fb6927fc640 1 -- 172.21.15.83:0/2356941783 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb680004eb0 con 0x7fb69c1409d0 2024-03-25T19:24:27.134 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.134+0000 7fb6927fc640 1 -- 172.21.15.83:0/2356941783 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2231476763 0 0) 0x7fb69c1438b0 con 0x7fb69c06adf0 2024-03-25T19:24:27.134 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.134+0000 7fb6927fc640 10 monclient(hunting): _negotiate 2024-03-25T19:24:27.134 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.134+0000 7fb6927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:27.134 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.134+0000 7fb6927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:27.134 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.134+0000 7fb6927fc640 10 monclient(hunting): my global_id is 4441 2024-03-25T19:24:27.134 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.134+0000 7fb6927fc640 1 -- 172.21.15.83:0/2356941783 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb69c142e30 con 0x7fb69c06adf0 2024-03-25T19:24:27.154 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.155+0000 7fb6927fc640 1 -- 172.21.15.83:0/2356941783 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3984297757 0 0) 0x7fb69c143d30 con 0x7fb69c0af580 2024-03-25T19:24:27.154 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.155+0000 7fb6927fc640 10 monclient(hunting): _negotiate 2024-03-25T19:24:27.155 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.155+0000 7fb6927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:27.155 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.155+0000 7fb6927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:27.155 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.155+0000 7fb6927fc640 10 monclient(hunting): my global_id is 4448 2024-03-25T19:24:27.155 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.155+0000 7fb6927fc640 1 -- 172.21.15.83:0/2356941783 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb69c1438b0 con 0x7fb69c0af580 2024-03-25T19:24:27.157 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.157+0000 7fb6927fc640 1 -- 172.21.15.83:0/2356941783 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 47595101 0 0) 0x7fb69c142e30 con 0x7fb69c06adf0 2024-03-25T19:24:27.157 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.157+0000 7fb6927fc640 1 -- 172.21.15.83:0/2356941783 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb69c143d30 con 0x7fb69c06adf0 2024-03-25T19:24:27.157 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.157+0000 7fb6927fc640 1 -- 172.21.15.83:0/2356941783 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fb6980019f0 con 0x7fb69c06adf0 2024-03-25T19:24:27.157 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.157+0000 7fb6927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:27.157 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.157+0000 7fb6927fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:24:27.157 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.157+0000 7fb6927fc640 10 monclient(hunting): dump: 2024-03-25T19:24:27.157 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:24:27.157 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:24:27.157 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:27.158 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:27.158 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:27.158 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:24:27.158 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:24:27.158 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:24:27.158 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:24:27.158 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:27.158 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.157+0000 7fb6927fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:24:27.158 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.159+0000 7fb6927fc640 1 -- 172.21.15.83:0/2356941783 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1923608837 0 0) 0x7fb680004eb0 con 0x7fb69c1409d0 2024-03-25T19:24:27.159 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.159+0000 7fb6927fc640 1 -- 172.21.15.83:0/2356941783 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb69c142e30 con 0x7fb69c1409d0 2024-03-25T19:24:27.159 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.159+0000 7fb6927fc640 1 -- 172.21.15.83:0/2356941783 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fb694001af0 con 0x7fb69c1409d0 2024-03-25T19:24:27.159 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.159+0000 7fb6927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:27.159 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.159+0000 7fb6927fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:24:27.159 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.159+0000 7fb6927fc640 10 monclient(hunting): dump: 2024-03-25T19:24:27.159 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:24:27.159 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:24:27.159 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:27.159 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:27.159 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:27.159 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:24:27.159 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:24:27.159 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:24:27.159 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:24:27.159 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:27.172 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.172+0000 7fb6927fc640 1 -- 172.21.15.83:0/2356941783 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1645057063 0 0) 0x7fb69c142e30 con 0x7fb69c1409d0 2024-03-25T19:24:27.173 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.172+0000 7fb6927fc640 1 -- 172.21.15.83:0/2356941783 >> v1:172.21.15.83:6790/0 conn(0x7fb69c0af580 legacy=0x7fb69c134e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:27.173 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.172+0000 7fb6927fc640 1 -- 172.21.15.83:0/2356941783 >> v1:172.21.15.103:6789/0 conn(0x7fb69c06adf0 legacy=0x7fb69c1430c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:27.173 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.172+0000 7fb6927fc640 10 monclient: _finish_hunting 0 2024-03-25T19:24:27.173 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.172+0000 7fb6927fc640 1 monclient: found mon.a 2024-03-25T19:24:27.173 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.172+0000 7fb6927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:27.173 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.172+0000 7fb6927fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:24:27.173 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.172+0000 7fb6927fc640 1 -- 172.21.15.83:0/2356941783 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb69c143ec0 con 0x7fb69c1409d0 2024-03-25T19:24:27.173 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.172+0000 7fb6927fc640 10 monclient: _finish_auth 0 2024-03-25T19:24:27.173 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.172+0000 7fb6927fc640 10 monclient: _check_auth_tickets 2024-03-25T19:24:27.173 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.172+0000 7fb6927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:27.173 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.172+0000 7fb6a3ff4640 5 monclient: authenticate success, global_id 4434 2024-03-25T19:24:27.173 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.172+0000 7fb6a3ff4640 10 monclient: _renew_subs 2024-03-25T19:24:27.173 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.172+0000 7fb6a3ff4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:24:27.173 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.172+0000 7fb6a3ff4640 1 -- 172.21.15.83:0/2356941783 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb69c135540 con 0x7fb69c1409d0 2024-03-25T19:24:27.173 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.172+0000 7fb6a3ff4640 10 client.4434.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:24:27.173 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.172+0000 7fb6a3ff4640 10 monclient: _renew_subs 2024-03-25T19:24:27.173 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.172+0000 7fb6a3ff4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:24:27.174 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.172+0000 7fb6a3ff4640 1 -- 172.21.15.83:0/2356941783 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb69c1390e0 con 0x7fb69c1409d0 2024-03-25T19:24:27.174 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.172+0000 7fb6a3ff4640 1 librados: init done 2024-03-25T19:24:27.174 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.173+0000 7fb6a3ff4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:27.174 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.173+0000 7fb6a3ff4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:27.174 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.173+0000 7fb6a3ff4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:24:27.174 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.173+0000 7fb6a3ff4640 1 -- 172.21.15.83:0/2356941783 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb69c143a40 con 0x7fb69c1409d0 2024-03-25T19:24:27.480 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.481+0000 7fb6927fc640 1 -- 172.21.15.83:0/2356941783 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb694001fe0 con 0x7fb69c1409d0 2024-03-25T19:24:27.480 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.481+0000 7fb6927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:27.480 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.481+0000 7fb6927fc640 1 -- 172.21.15.83:0/2356941783 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fb694003b30 con 0x7fb69c1409d0 2024-03-25T19:24:27.480 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.481+0000 7fb6927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:27.481 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.481+0000 7fb6927fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:24:27.481 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.481+0000 7fb6927fc640 10 monclient: dump: 2024-03-25T19:24:27.481 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:24:27.481 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:24:27.481 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:27.481 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:27.481 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:27.481 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:24:27.481 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:24:27.481 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:24:27.481 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:24:27.481 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:27.481 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.481+0000 7fb693fff640 4 set_mon_vals no callback set 2024-03-25T19:24:27.482 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.481+0000 7fb6927fc640 1 -- 172.21.15.83:0/2356941783 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7fb694019b00 con 0x7fb69c1409d0 2024-03-25T19:24:27.482 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.482+0000 7fb6a1d90640 10 monclient: get_auth_request con 0x7fb68006df70 auth_method 0 2024-03-25T19:24:27.482 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.482+0000 7fb6927fc640 10 client.4434.objecter ms_handle_connect 0x7fb68006df70 2024-03-25T19:24:27.489 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.489+0000 7fb6927fc640 1 -- 172.21.15.83:0/2356941783 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7fb694053860 con 0x7fb69c1409d0 2024-03-25T19:24:27.489 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.489+0000 7fb6927fc640 10 client.4434.objecter ms_dispatch 0x7fb69c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:24:27.489 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.489+0000 7fb6927fc640 3 client.4434.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:24:27.489 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.489+0000 7fb6927fc640 3 client.4434.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:24:27.489 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.489+0000 7fb6927fc640 20 client.4434.objecter dump_active .. 0 homeless 2024-03-25T19:24:27.489 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.489+0000 7fb6927fc640 1 -- 172.21.15.83:0/2356941783 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb69404f1a0 con 0x7fb69c1409d0 2024-03-25T19:24:27.489 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.489+0000 7fb6927fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:27.489 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.489+0000 7fb6927fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:24:27.703 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.703+0000 7fb6a3ff4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:27.703 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.703+0000 7fb6a3ff4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:27.703 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.703+0000 7fb6a3ff4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:24:27.703 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.703+0000 7fb6a3ff4640 1 -- 172.21.15.83:0/2356941783 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb69c0627b0 con 0x7fb69c1409d0 2024-03-25T19:24:27.792 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.792+0000 7fb6927fc640 1 -- 172.21.15.83:0/2356941783 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7fb6940512a0 con 0x7fb69c1409d0 2024-03-25T19:24:27.792 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.792+0000 7fb6927fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:27.792 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.792+0000 7fb6927fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:24:27.792 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:24:27.792 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:24:27.795 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.795+0000 7fb6a3ff4640 10 librados: watch_flush enter 2024-03-25T19:24:27.795 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.795+0000 7fb6a3ff4640 10 librados: watch_flush exit 2024-03-25T19:24:27.795 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.795+0000 7fb6a3ff4640 20 client.4434.objecter shutdown clearing up homeless session... 2024-03-25T19:24:27.795 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.795+0000 7fb6a3ff4640 10 client.4434.objecter successfully canceled tick 2024-03-25T19:24:27.795 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.795+0000 7fb6a3ff4640 1 -- 172.21.15.83:0/2356941783 >> v1:172.21.15.103:6800/345072327 conn(0x7fb68006df70 legacy=0x7fb680070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:27.795 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.795+0000 7fb6a3ff4640 10 monclient: shutdown 2024-03-25T19:24:27.795 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.795+0000 7fb6a3ff4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:27.795 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.795+0000 7fb6a3ff4640 1 -- 172.21.15.83:0/2356941783 >> v1:172.21.15.83:6789/0 conn(0x7fb69c1409d0 legacy=0x7fb69c1d5070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:27.795 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.795+0000 7fb6a3ff4640 1 -- 172.21.15.83:0/2356941783 shutdown_connections 2024-03-25T19:24:27.796 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.795+0000 7fb6a3ff4640 1 -- 172.21.15.83:0/2356941783 >> 172.21.15.83:0/2356941783 conn(0x7fb69c0b4b80 msgr2=0x7fb69c071850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:27.796 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.795+0000 7fb6a3ff4640 1 -- 172.21.15.83:0/2356941783 shutdown_connections 2024-03-25T19:24:27.796 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.795+0000 7fb6a3ff4640 1 -- 172.21.15.83:0/2356941783 wait complete. 2024-03-25T19:24:27.796 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:27.795+0000 7fb6a3ff4640 1 librados: shutdown 2024-03-25T19:24:27.805 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:24:34.611 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:24:34.726 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.727+0000 7fd303b0d640 10 monclient: get_monmap_and_config 2024-03-25T19:24:34.727 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.727+0000 7fd303b0d640 10 monclient: build_initial_monmap 2024-03-25T19:24:34.727 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.727+0000 7fd303b0d640 10 monclient: monmap: 2024-03-25T19:24:34.727 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:24:34.727 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:24:34.727 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:24:34.728126+0000 2024-03-25T19:24:34.727 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:24:34.728126+0000 2024-03-25T19:24:34.727 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:34.727 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:24:34.727 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:24:34.727 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:24:34.727 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:24:34.727 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:34.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.727+0000 7fd303b0d640 1 Processor -- start 2024-03-25T19:24:34.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.728+0000 7fd303b0d640 1 -- start start 2024-03-25T19:24:34.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.728+0000 7fd303b0d640 10 monclient: init 2024-03-25T19:24:34.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.728+0000 7fd303b0d640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:34.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.728+0000 7fd303b0d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:24:34.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.728+0000 7fd303b0d640 10 monclient(hunting): picked mon.noname-a con 0x7fd2fc0a98f0 addr v1:172.21.15.83:6789/0 2024-03-25T19:24:34.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.728+0000 7fd303b0d640 10 monclient(hunting): picked mon.noname-b con 0x7fd2fc13f710 addr v1:172.21.15.83:6790/0 2024-03-25T19:24:34.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.728+0000 7fd303b0d640 10 monclient(hunting): picked mon.noname-c con 0x7fd2fc1430c0 addr v1:172.21.15.103:6789/0 2024-03-25T19:24:34.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.728+0000 7fd303b0d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2fc142520 con 0x7fd2fc0a98f0 2024-03-25T19:24:34.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.728+0000 7fd303b0d640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2fc0b28a0 con 0x7fd2fc1430c0 2024-03-25T19:24:34.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.728+0000 7fd303b0d640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2fc0b2a30 con 0x7fd2fc13f710 2024-03-25T19:24:34.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.728+0000 7fd303b0d640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:34.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.728+0000 7fd303b0d640 10 monclient(hunting): authenticate will time out at 6332.000519s 2024-03-25T19:24:34.737 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.737+0000 7fd3010a8640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fd2fc13f710 0x7fd2fc1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:55642/0 (socket says 172.21.15.83:55642) 2024-03-25T19:24:34.737 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.737+0000 7fd3010a8640 1 -- 172.21.15.83:0/701917013 learned_addr learned my addr 172.21.15.83:0/701917013 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:24:34.750 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.750+0000 7fd3008a7640 1 -- 172.21.15.83:0/701917013 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2529118254 0 0) 0x7fd2fc0b2a30 con 0x7fd2fc13f710 2024-03-25T19:24:34.750 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.750+0000 7fd3008a7640 10 monclient(hunting): _negotiate 2024-03-25T19:24:34.750 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.750+0000 7fd3008a7640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:34.750 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.750+0000 7fd3008a7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:34.750 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.750+0000 7fd3008a7640 10 monclient(hunting): my global_id is 4451 2024-03-25T19:24:34.750 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.750+0000 7fd3008a7640 1 -- 172.21.15.83:0/701917013 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2e8003520 con 0x7fd2fc13f710 2024-03-25T19:24:34.755 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.756+0000 7fd3008a7640 1 -- 172.21.15.83:0/701917013 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2268534496 0 0) 0x7fd2fc142520 con 0x7fd2fc0a98f0 2024-03-25T19:24:34.755 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.756+0000 7fd3008a7640 10 monclient(hunting): _negotiate 2024-03-25T19:24:34.756 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.756+0000 7fd3008a7640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:34.756 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.756+0000 7fd3008a7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:34.756 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.756+0000 7fd3008a7640 10 monclient(hunting): my global_id is 4437 2024-03-25T19:24:34.756 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.756+0000 7fd3008a7640 1 -- 172.21.15.83:0/701917013 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2fc0b2a30 con 0x7fd2fc0a98f0 2024-03-25T19:24:34.761 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.761+0000 7fd3008a7640 1 -- 172.21.15.83:0/701917013 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3153068342 0 0) 0x7fd2fc0b28a0 con 0x7fd2fc1430c0 2024-03-25T19:24:34.761 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.761+0000 7fd3008a7640 10 monclient(hunting): _negotiate 2024-03-25T19:24:34.761 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.761+0000 7fd3008a7640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:34.761 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.761+0000 7fd3008a7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:34.761 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.761+0000 7fd3008a7640 10 monclient(hunting): my global_id is 4444 2024-03-25T19:24:34.761 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.761+0000 7fd3008a7640 1 -- 172.21.15.83:0/701917013 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2fc142520 con 0x7fd2fc1430c0 2024-03-25T19:24:34.763 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.763+0000 7fd3008a7640 1 -- 172.21.15.83:0/701917013 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1429417528 0 0) 0x7fd2e8003520 con 0x7fd2fc13f710 2024-03-25T19:24:34.763 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.763+0000 7fd3008a7640 1 -- 172.21.15.83:0/701917013 >> v1:172.21.15.103:6789/0 conn(0x7fd2fc1430c0 legacy=0x7fd2fc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:34.763 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.763+0000 7fd3008a7640 1 -- 172.21.15.83:0/701917013 >> v1:172.21.15.83:6789/0 conn(0x7fd2fc0a98f0 legacy=0x7fd2fc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:34.763 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.763+0000 7fd3008a7640 10 monclient: _finish_hunting 0 2024-03-25T19:24:34.763 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.763+0000 7fd3008a7640 1 monclient: found mon.noname-b 2024-03-25T19:24:34.764 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.763+0000 7fd3008a7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:34.764 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.763+0000 7fd3008a7640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.83:6790/0 2024-03-25T19:24:34.764 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.763+0000 7fd3008a7640 1 -- 172.21.15.83:0/701917013 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2fc0b44c0 con 0x7fd2fc13f710 2024-03-25T19:24:34.764 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.763+0000 7fd3008a7640 10 monclient: _finish_auth 0 2024-03-25T19:24:34.764 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.763+0000 7fd3008a7640 10 monclient: _check_auth_tickets 2024-03-25T19:24:34.764 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.763+0000 7fd3008a7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:34.764 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.763+0000 7fd3008a7640 1 -- 172.21.15.83:0/701917013 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fd2ec001690 con 0x7fd2fc13f710 2024-03-25T19:24:34.764 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.763+0000 7fd3008a7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:34.764 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.763+0000 7fd3008a7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:24:34.764 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.763+0000 7fd3008a7640 10 monclient: dump: 2024-03-25T19:24:34.764 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:24:34.764 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:24:34.764 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:34.764 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:34.764 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:34.765 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:24:34.765 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:24:34.765 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:24:34.765 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:24:34.765 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:34.765 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.763+0000 7fd303b0d640 5 monclient: authenticate success, global_id 4451 2024-03-25T19:24:34.765 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.763+0000 7fd303b0d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:24:34.781 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.781+0000 7fd3008a7640 1 -- 172.21.15.83:0/701917013 <== mon.2 v1:172.21.15.83:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2ec0017f0 con 0x7fd2fc13f710 2024-03-25T19:24:34.781 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.781+0000 7fd3008a7640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:34.781 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.781+0000 7fd3008a7640 1 -- 172.21.15.83:0/701917013 <== mon.2 v1:172.21.15.83:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fd2ec001c00 con 0x7fd2fc13f710 2024-03-25T19:24:34.781 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.781+0000 7fd3008a7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:34.781 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.781+0000 7fd3008a7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:24:34.781 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.781+0000 7fd3008a7640 10 monclient: dump: 2024-03-25T19:24:34.782 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:24:34.782 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:24:34.782 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:34.782 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:34.782 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:34.782 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:24:34.782 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:24:34.782 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:24:34.782 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:24:34.782 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:34.782 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.781+0000 7fd303b0d640 10 monclient: get_monmap_and_config success 2024-03-25T19:24:34.782 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.781+0000 7fd303b0d640 4 set_mon_vals no callback set 2024-03-25T19:24:34.784 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.782+0000 7fd303b0d640 10 monclient: shutdown 2024-03-25T19:24:34.784 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.782+0000 7fd303b0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:34.784 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.782+0000 7fd303b0d640 1 -- 172.21.15.83:0/701917013 >> v1:172.21.15.83:6790/0 conn(0x7fd2fc13f710 legacy=0x7fd2fc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:34.785 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.782+0000 7fd303b0d640 1 -- 172.21.15.83:0/701917013 shutdown_connections 2024-03-25T19:24:34.785 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.782+0000 7fd303b0d640 1 -- 172.21.15.83:0/701917013 >> 172.21.15.83:0/701917013 conn(0x7fd2fc0b7ea0 msgr2=0x7fd2fc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:34.785 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.782+0000 7fd303b0d640 1 -- 172.21.15.83:0/701917013 shutdown_connections 2024-03-25T19:24:34.785 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.782+0000 7fd303b0d640 1 -- 172.21.15.83:0/701917013 wait complete. 2024-03-25T19:24:34.785 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.783+0000 7fd303b0d640 10 monclient: build_initial_monmap 2024-03-25T19:24:34.785 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.783+0000 7fd303b0d640 10 monclient: monmap: 2024-03-25T19:24:34.785 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:24:34.785 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:24:34.785 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:24:34.785 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:24:34.785 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:34.785 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:24:34.785 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:24:34.785 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:24:34.785 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:24:34.785 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:34.786 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.783+0000 7fd303b0d640 1 librados: starting msgr at 2024-03-25T19:24:34.786 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.783+0000 7fd303b0d640 1 librados: starting objecter 2024-03-25T19:24:34.786 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.784+0000 7fd303b0d640 1 Processor -- start 2024-03-25T19:24:34.786 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.784+0000 7fd303b0d640 1 -- start start 2024-03-25T19:24:34.786 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.784+0000 7fd303b0d640 1 librados: setting wanted keys 2024-03-25T19:24:34.786 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.784+0000 7fd303b0d640 1 librados: calling monclient init 2024-03-25T19:24:34.786 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.784+0000 7fd303b0d640 10 monclient: init 2024-03-25T19:24:34.786 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.784+0000 7fd303b0d640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:34.786 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.784+0000 7fd303b0d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:24:34.786 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.784+0000 7fd303b0d640 10 monclient(hunting): picked mon.noname-b con 0x7fd2fc0a98f0 addr v1:172.21.15.103:6789/0 2024-03-25T19:24:34.786 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.784+0000 7fd303b0d640 10 monclient(hunting): picked mon.noname-a con 0x7fd2fc13f710 addr v1:172.21.15.83:6789/0 2024-03-25T19:24:34.786 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.784+0000 7fd303b0d640 10 monclient(hunting): picked mon.noname-c con 0x7fd2fc1430c0 addr v1:172.21.15.83:6790/0 2024-03-25T19:24:34.786 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.784+0000 7fd303b0d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2fc0abc30 con 0x7fd2fc13f710 2024-03-25T19:24:34.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.785+0000 7fd303b0d640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2fc0abf10 con 0x7fd2fc0a98f0 2024-03-25T19:24:34.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.785+0000 7fd303b0d640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2fc138fe0 con 0x7fd2fc1430c0 2024-03-25T19:24:34.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.785+0000 7fd303b0d640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:34.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.785+0000 7fd303b0d640 10 monclient(hunting): authenticate will time out at 6332.056820s 2024-03-25T19:24:34.798 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.798+0000 7fd3010a8640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fd2fc13f710 0x7fd2fc1387e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39334/0 (socket says 172.21.15.83:39334) 2024-03-25T19:24:34.798 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.798+0000 7fd3010a8640 1 -- 172.21.15.83:0/442353949 learned_addr learned my addr 172.21.15.83:0/442353949 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:24:34.800 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.800+0000 7fd2f27fc640 10 client.?.objecter ms_handle_connect 0x7fd2fc13f710 2024-03-25T19:24:34.800 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.800+0000 7fd2f27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:34.807 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.808+0000 7fd2f27fc640 1 -- 172.21.15.83:0/442353949 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3807091876 0 0) 0x7fd2fc0abc30 con 0x7fd2fc13f710 2024-03-25T19:24:34.807 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.808+0000 7fd2f27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:24:34.807 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.808+0000 7fd2f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:34.808 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.808+0000 7fd2f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:34.808 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.808+0000 7fd2f27fc640 10 monclient(hunting): my global_id is 4440 2024-03-25T19:24:34.808 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.808+0000 7fd2f27fc640 1 -- 172.21.15.83:0/442353949 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2e0004eb0 con 0x7fd2fc13f710 2024-03-25T19:24:34.808 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.808+0000 7fd2f27fc640 10 client.?.objecter ms_handle_connect 0x7fd2fc1430c0 2024-03-25T19:24:34.808 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.808+0000 7fd2f27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:34.809 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.810+0000 7fd2f27fc640 10 client.?.objecter ms_handle_connect 0x7fd2fc0a98f0 2024-03-25T19:24:34.809 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.810+0000 7fd2f27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:34.812 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.813+0000 7fd2f27fc640 1 -- 172.21.15.83:0/442353949 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2965964525 0 0) 0x7fd2fc138fe0 con 0x7fd2fc1430c0 2024-03-25T19:24:34.812 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.813+0000 7fd2f27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:24:34.812 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.813+0000 7fd2f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:34.812 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.813+0000 7fd2f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:34.813 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.813+0000 7fd2f27fc640 10 monclient(hunting): my global_id is 4454 2024-03-25T19:24:34.813 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.813+0000 7fd2f27fc640 1 -- 172.21.15.83:0/442353949 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2fc0abc30 con 0x7fd2fc1430c0 2024-03-25T19:24:34.818 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.818+0000 7fd2f27fc640 1 -- 172.21.15.83:0/442353949 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3819132296 0 0) 0x7fd2fc0abc30 con 0x7fd2fc1430c0 2024-03-25T19:24:34.818 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.818+0000 7fd2f27fc640 1 -- 172.21.15.83:0/442353949 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2fc138fe0 con 0x7fd2fc1430c0 2024-03-25T19:24:34.818 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.818+0000 7fd2f27fc640 1 -- 172.21.15.83:0/442353949 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fd2f4001b50 con 0x7fd2fc1430c0 2024-03-25T19:24:34.818 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.818+0000 7fd2f27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:34.819 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.818+0000 7fd2f27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:24:34.819 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.818+0000 7fd2f27fc640 10 monclient(hunting): dump: 2024-03-25T19:24:34.819 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:24:34.819 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:24:34.819 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:34.819 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:34.819 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:34.819 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:24:34.819 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:24:34.819 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:24:34.819 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:24:34.819 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:34.819 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.818+0000 7fd2f27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:24:34.823 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.824+0000 7fd2f27fc640 1 -- 172.21.15.83:0/442353949 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2523675885 0 0) 0x7fd2e0004eb0 con 0x7fd2fc13f710 2024-03-25T19:24:34.823 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.824+0000 7fd2f27fc640 1 -- 172.21.15.83:0/442353949 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2fc0abc30 con 0x7fd2fc13f710 2024-03-25T19:24:34.823 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.824+0000 7fd2f27fc640 1 -- 172.21.15.83:0/442353949 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fd2ec001bc0 con 0x7fd2fc13f710 2024-03-25T19:24:34.824 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.824+0000 7fd2f27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:34.824 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.824+0000 7fd2f27fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:24:34.824 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.824+0000 7fd2f27fc640 10 monclient(hunting): dump: 2024-03-25T19:24:34.824 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:24:34.824 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:24:34.825 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:34.825 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:34.825 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:34.825 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:24:34.825 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:24:34.825 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:24:34.825 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:24:34.826 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:34.827 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.824+0000 7fd2f27fc640 1 -- 172.21.15.83:0/442353949 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3169124469 0 0) 0x7fd2fc0abf10 con 0x7fd2fc0a98f0 2024-03-25T19:24:34.827 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.824+0000 7fd2f27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:24:34.827 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.824+0000 7fd2f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:34.827 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.824+0000 7fd2f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:34.827 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.824+0000 7fd2f27fc640 10 monclient(hunting): my global_id is 4447 2024-03-25T19:24:34.827 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.824+0000 7fd2f27fc640 1 -- 172.21.15.83:0/442353949 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2e0004eb0 con 0x7fd2fc0a98f0 2024-03-25T19:24:34.829 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.829+0000 7fd2f27fc640 1 -- 172.21.15.83:0/442353949 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1644115117 0 0) 0x7fd2fc138fe0 con 0x7fd2fc1430c0 2024-03-25T19:24:34.829 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.829+0000 7fd2f27fc640 1 -- 172.21.15.83:0/442353949 >> v1:172.21.15.103:6789/0 conn(0x7fd2fc0a98f0 legacy=0x7fd2fc136540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:34.829 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.829+0000 7fd2f27fc640 1 -- 172.21.15.83:0/442353949 >> v1:172.21.15.83:6789/0 conn(0x7fd2fc13f710 legacy=0x7fd2fc1387e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:34.829 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.829+0000 7fd2f27fc640 10 monclient: _finish_hunting 0 2024-03-25T19:24:34.829 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.829+0000 7fd2f27fc640 1 monclient: found mon.c 2024-03-25T19:24:34.830 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.829+0000 7fd2f27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:34.830 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.829+0000 7fd2f27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:24:34.830 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.829+0000 7fd2f27fc640 1 -- 172.21.15.83:0/442353949 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2fc137bf0 con 0x7fd2fc1430c0 2024-03-25T19:24:34.830 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.829+0000 7fd2f27fc640 10 monclient: _finish_auth 0 2024-03-25T19:24:34.830 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.829+0000 7fd2f27fc640 10 monclient: _check_auth_tickets 2024-03-25T19:24:34.830 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.829+0000 7fd2f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:34.830 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.829+0000 7fd303b0d640 5 monclient: authenticate success, global_id 4454 2024-03-25T19:24:34.830 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.829+0000 7fd303b0d640 10 monclient: _renew_subs 2024-03-25T19:24:34.830 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.829+0000 7fd303b0d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:24:34.830 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.829+0000 7fd303b0d640 1 -- 172.21.15.83:0/442353949 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2fc137e40 con 0x7fd2fc1430c0 2024-03-25T19:24:34.830 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.829+0000 7fd303b0d640 10 client.4454.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:24:34.830 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.829+0000 7fd303b0d640 10 monclient: _renew_subs 2024-03-25T19:24:34.830 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.829+0000 7fd303b0d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:24:34.830 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.829+0000 7fd303b0d640 1 -- 172.21.15.83:0/442353949 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2fc1e3690 con 0x7fd2fc1430c0 2024-03-25T19:24:34.831 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.829+0000 7fd303b0d640 1 librados: init done 2024-03-25T19:24:34.831 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.830+0000 7fd303b0d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:34.831 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.830+0000 7fd303b0d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:34.831 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.830+0000 7fd303b0d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:24:34.831 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:34.830+0000 7fd303b0d640 1 -- 172.21.15.83:0/442353949 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2fc0627b0 con 0x7fd2fc1430c0 2024-03-25T19:24:35.051 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:35.051+0000 7fd2f27fc640 1 -- 172.21.15.83:0/442353949 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2f4001fe0 con 0x7fd2fc1430c0 2024-03-25T19:24:35.051 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:35.051+0000 7fd2f27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:35.051 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:35.051+0000 7fd2f27fc640 1 -- 172.21.15.83:0/442353949 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fd2f4003b30 con 0x7fd2fc1430c0 2024-03-25T19:24:35.051 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:35.051+0000 7fd2f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:35.051 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:35.051+0000 7fd2f27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:24:35.051 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:35.051+0000 7fd2f27fc640 10 monclient: dump: 2024-03-25T19:24:35.051 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:24:35.051 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:24:35.051 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:35.051 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:35.051 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:35.051 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:24:35.051 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:24:35.051 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:24:35.052 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:24:35.052 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:35.052 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:35.051+0000 7fd2f3fff640 4 set_mon_vals no callback set 2024-03-25T19:24:35.133 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:35.133+0000 7fd2f27fc640 1 -- 172.21.15.83:0/442353949 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7fd2f4003c90 con 0x7fd2fc1430c0 2024-03-25T19:24:35.133 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:35.134+0000 7fd3018a9640 10 monclient: get_auth_request con 0x7fd2e006df50 auth_method 0 2024-03-25T19:24:35.133 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:35.134+0000 7fd2f27fc640 10 client.4454.objecter ms_handle_connect 0x7fd2e006df50 2024-03-25T19:24:35.142 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:35.142+0000 7fd2f27fc640 1 -- 172.21.15.83:0/442353949 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7fd2f40537e0 con 0x7fd2fc1430c0 2024-03-25T19:24:35.142 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:35.142+0000 7fd2f27fc640 10 client.4454.objecter ms_dispatch 0x7fd2fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:24:35.142 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:35.142+0000 7fd2f27fc640 3 client.4454.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:24:35.142 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:35.142+0000 7fd2f27fc640 3 client.4454.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:24:35.142 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:35.142+0000 7fd2f27fc640 20 client.4454.objecter dump_active .. 0 homeless 2024-03-25T19:24:35.143 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:35.142+0000 7fd2f27fc640 1 -- 172.21.15.83:0/442353949 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd2f404f120 con 0x7fd2fc1430c0 2024-03-25T19:24:35.143 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:35.142+0000 7fd2f27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:35.143 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:35.142+0000 7fd2f27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:24:35.364 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:35.364+0000 7fd303b0d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:35.364 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:35.364+0000 7fd303b0d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:35.364 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:35.364+0000 7fd303b0d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:24:35.364 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:35.364+0000 7fd303b0d640 1 -- 172.21.15.83:0/442353949 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd2fc13e250 con 0x7fd2fc1430c0 2024-03-25T19:24:35.390 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:35.391+0000 7fd2f27fc640 1 -- 172.21.15.83:0/442353949 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7fd2f4051220 con 0x7fd2fc1430c0 2024-03-25T19:24:35.391 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:35.391+0000 7fd2f27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:35.391 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:35.391+0000 7fd2f27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:24:35.391 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:24:35.391 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:24:35.394 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:35.393+0000 7fd303b0d640 10 librados: watch_flush enter 2024-03-25T19:24:35.394 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:35.393+0000 7fd303b0d640 10 librados: watch_flush exit 2024-03-25T19:24:35.394 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:35.393+0000 7fd303b0d640 20 client.4454.objecter shutdown clearing up homeless session... 2024-03-25T19:24:35.394 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:35.393+0000 7fd303b0d640 10 client.4454.objecter successfully canceled tick 2024-03-25T19:24:35.394 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:35.393+0000 7fd303b0d640 1 -- 172.21.15.83:0/442353949 >> v1:172.21.15.103:6800/345072327 conn(0x7fd2e006df50 legacy=0x7fd2e0070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:35.394 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:35.393+0000 7fd303b0d640 10 monclient: shutdown 2024-03-25T19:24:35.394 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:35.393+0000 7fd303b0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:35.394 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:35.393+0000 7fd303b0d640 1 -- 172.21.15.83:0/442353949 >> v1:172.21.15.83:6790/0 conn(0x7fd2fc1430c0 legacy=0x7fd2fc0ac330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:35.394 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:35.393+0000 7fd303b0d640 1 -- 172.21.15.83:0/442353949 shutdown_connections 2024-03-25T19:24:35.394 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:35.393+0000 7fd303b0d640 1 -- 172.21.15.83:0/442353949 >> 172.21.15.83:0/442353949 conn(0x7fd2fc0b7ea0 msgr2=0x7fd2fc13c590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:35.395 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:35.394+0000 7fd303b0d640 1 -- 172.21.15.83:0/442353949 shutdown_connections 2024-03-25T19:24:35.395 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:35.394+0000 7fd303b0d640 1 -- 172.21.15.83:0/442353949 wait complete. 2024-03-25T19:24:35.395 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:35.394+0000 7fd303b0d640 1 librados: shutdown 2024-03-25T19:24:35.406 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:24:42.212 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:24:42.322 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.322+0000 7f104a474640 10 monclient: get_monmap_and_config 2024-03-25T19:24:42.322 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.322+0000 7f104a474640 10 monclient: build_initial_monmap 2024-03-25T19:24:42.322 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.322+0000 7f104a474640 10 monclient: monmap: 2024-03-25T19:24:42.322 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:24:42.323 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:24:42.323 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:24:42.323718+0000 2024-03-25T19:24:42.323 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:24:42.323718+0000 2024-03-25T19:24:42.323 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:42.323 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:24:42.323 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:24:42.323 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:24:42.323 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:24:42.323 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:42.324 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.323+0000 7f104a474640 1 Processor -- start 2024-03-25T19:24:42.324 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.323+0000 7f104a474640 1 -- start start 2024-03-25T19:24:42.324 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.323+0000 7f104a474640 10 monclient: init 2024-03-25T19:24:42.324 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.324+0000 7f104a474640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:42.324 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.324+0000 7f104a474640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:24:42.324 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.324+0000 7f104a474640 10 monclient(hunting): picked mon.noname-a con 0x7f10440a98f0 addr v1:172.21.15.83:6789/0 2024-03-25T19:24:42.324 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.324+0000 7f104a474640 10 monclient(hunting): picked mon.noname-c con 0x7f104413f710 addr v1:172.21.15.103:6789/0 2024-03-25T19:24:42.325 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.324+0000 7f104a474640 10 monclient(hunting): picked mon.noname-b con 0x7f10441430c0 addr v1:172.21.15.83:6790/0 2024-03-25T19:24:42.325 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.324+0000 7f104a474640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1044142520 con 0x7f10440a98f0 2024-03-25T19:24:42.325 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.324+0000 7f104a474640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10440b28a0 con 0x7f104413f710 2024-03-25T19:24:42.325 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.324+0000 7f104a474640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10440b2a30 con 0x7f10441430c0 2024-03-25T19:24:42.325 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.324+0000 7f104a474640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:42.325 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.324+0000 7f104a474640 10 monclient(hunting): authenticate will time out at 6339.596077s 2024-03-25T19:24:42.327 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.328+0000 7f10437fe640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f104413f710 0x7f10441429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:42716/0 (socket says 172.21.15.83:42716) 2024-03-25T19:24:42.328 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.328+0000 7f10437fe640 1 -- 172.21.15.83:0/1407112227 learned_addr learned my addr 172.21.15.83:0/1407112227 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:24:42.332 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.333+0000 7f1042ffd640 1 -- 172.21.15.83:0/1407112227 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 311298908 0 0) 0x7f10440b28a0 con 0x7f104413f710 2024-03-25T19:24:42.332 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.333+0000 7f1042ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:24:42.332 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.333+0000 7f1042ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:42.332 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.333+0000 7f1042ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:42.333 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.333+0000 7f1042ffd640 10 monclient(hunting): my global_id is 4450 2024-03-25T19:24:42.333 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.333+0000 7f1042ffd640 1 -- 172.21.15.83:0/1407112227 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f102c003520 con 0x7f104413f710 2024-03-25T19:24:42.340 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.340+0000 7f1042ffd640 1 -- 172.21.15.83:0/1407112227 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3586882806 0 0) 0x7f102c003520 con 0x7f104413f710 2024-03-25T19:24:42.340 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.340+0000 7f1042ffd640 1 -- 172.21.15.83:0/1407112227 >> v1:172.21.15.83:6790/0 conn(0x7f10441430c0 legacy=0x7f104414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:42.340 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.340+0000 7f1042ffd640 1 -- 172.21.15.83:0/1407112227 >> v1:172.21.15.83:6789/0 conn(0x7f10440a98f0 legacy=0x7f10440a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:24:42.340 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.340+0000 7f1042ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:24:42.341 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.340+0000 7f1042ffd640 1 monclient: found mon.noname-c 2024-03-25T19:24:42.341 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.340+0000 7f1042ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:42.341 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.340+0000 7f1042ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.103:6789/0 2024-03-25T19:24:42.341 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.340+0000 7f1042ffd640 1 -- 172.21.15.83:0/1407112227 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10440b44c0 con 0x7f104413f710 2024-03-25T19:24:42.341 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.340+0000 7f1042ffd640 10 monclient: _finish_auth 0 2024-03-25T19:24:42.341 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.340+0000 7f1042ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:24:42.341 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.340+0000 7f1042ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:42.341 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.340+0000 7f1042ffd640 1 -- 172.21.15.83:0/1407112227 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f1038001690 con 0x7f104413f710 2024-03-25T19:24:42.341 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.340+0000 7f104a474640 5 monclient: authenticate success, global_id 4450 2024-03-25T19:24:42.341 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.340+0000 7f104a474640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:24:42.341 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.340+0000 7f1042ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:42.341 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.340+0000 7f1042ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:24:42.341 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.340+0000 7f1042ffd640 10 monclient: dump: 2024-03-25T19:24:42.341 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:24:42.341 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:24:42.341 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:42.342 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:42.342 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:42.342 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:24:42.342 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:24:42.342 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:24:42.342 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:24:42.342 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:42.342 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.340+0000 7f104a474640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:24:42.343 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.342+0000 7f10437fe640 1 -- 172.21.15.83:0/1407112227 >> v1:172.21.15.103:6789/0 conn(0x7f104413f710 legacy=0x7f10441429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-03-25T19:24:42.343 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.342+0000 7f10437fe640 1 -- 172.21.15.83:0/1407112227 >> v1:172.21.15.103:6789/0 conn(0x7f104413f710 legacy=0x7f10441429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:24:42.343 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.342+0000 7f10437fe640 1 --1- 172.21.15.83:0/1407112227 >> v1:172.21.15.103:6789/0 conn(0x7f104413f710 0x7f10441429b0 :-1 s=OPENED pgs=33864 cs=1 l=1).handle_message read tag failed 2024-03-25T19:24:42.343 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.342+0000 7f10437fe640 1 --1- 172.21.15.83:0/1407112227 >> v1:172.21.15.103:6789/0 conn(0x7f104413f710 0x7f10441429b0 :-1 s=OPENED pgs=33864 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:24:42.343 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.342+0000 7f1042ffd640 10 monclient: ms_handle_reset current mon v1:172.21.15.103:6789/0 2024-03-25T19:24:42.343 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.342+0000 7f1042ffd640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:42.343 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.342+0000 7f1042ffd640 1 -- 172.21.15.83:0/1407112227 >> v1:172.21.15.103:6789/0 conn(0x7f104413f710 legacy=0x7f10441429b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:24:42.343 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.342+0000 7f1042ffd640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:24:42.343 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.342+0000 7f1042ffd640 10 monclient(hunting): picked mon.b con 0x7f102c003aa0 addr v1:172.21.15.103:6789/0 2024-03-25T19:24:42.343 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.342+0000 7f1042ffd640 10 monclient(hunting): picked mon.a con 0x7f10440a98f0 addr v1:172.21.15.83:6789/0 2024-03-25T19:24:42.343 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.342+0000 7f1042ffd640 10 monclient(hunting): picked mon.c con 0x7f10441430c0 addr v1:172.21.15.83:6790/0 2024-03-25T19:24:42.343 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.342+0000 7f1042ffd640 1 -- 172.21.15.83:0/1407112227 --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f102c003520 con 0x7f10440a98f0 2024-03-25T19:24:42.343 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.342+0000 7f1042ffd640 1 -- 172.21.15.83:0/1407112227 --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1044142520 con 0x7f102c003aa0 2024-03-25T19:24:42.344 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.342+0000 7f1042ffd640 1 -- 172.21.15.83:0/1407112227 --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f10440b28a0 con 0x7f10441430c0 2024-03-25T19:24:42.344 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.342+0000 7f1042ffd640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:42.344 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.342+0000 7f1042ffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.103:6789/0 2024-03-25T19:24:42.353 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.354+0000 7f1042ffd640 1 -- 172.21.15.83:0/1407112227 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1172159560 0 0) 0x7f10440b28a0 con 0x7f10441430c0 2024-03-25T19:24:42.353 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.354+0000 7f1042ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:24:42.353 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.354+0000 7f1042ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:42.353 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.354+0000 7f1042ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-25T19:24:42.354 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.354+0000 7f1042ffd640 1 -- 172.21.15.83:0/1407112227 --> v1:172.21.15.83:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f102c00ecc0 con 0x7f10441430c0 2024-03-25T19:24:42.363 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.363+0000 7f1042ffd640 1 -- 172.21.15.83:0/1407112227 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2385007906 0 0) 0x7f1044142520 con 0x7f102c003aa0 2024-03-25T19:24:42.363 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.363+0000 7f1042ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:24:42.363 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.363+0000 7f1042ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:42.363 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.363+0000 7f1042ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-25T19:24:42.363 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.363+0000 7f1042ffd640 1 -- 172.21.15.83:0/1407112227 --> v1:172.21.15.103:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f10440b28a0 con 0x7f102c003aa0 2024-03-25T19:24:42.365 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.365+0000 7f1042ffd640 1 -- 172.21.15.83:0/1407112227 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1419369065 0 0) 0x7f102c003520 con 0x7f10440a98f0 2024-03-25T19:24:42.365 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.365+0000 7f1042ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:24:42.365 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.365+0000 7f1042ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:42.365 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.365+0000 7f1042ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-25T19:24:42.365 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.365+0000 7f1042ffd640 1 -- 172.21.15.83:0/1407112227 --> v1:172.21.15.83:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f1044142520 con 0x7f10440a98f0 2024-03-25T19:24:42.369 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.369+0000 7f1042ffd640 1 -- 172.21.15.83:0/1407112227 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 1543562956 0 0) 0x7f10440b28a0 con 0x7f102c003aa0 2024-03-25T19:24:42.369 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.369+0000 7f1042ffd640 1 -- 172.21.15.83:0/1407112227 >> v1:172.21.15.83:6790/0 conn(0x7f10441430c0 legacy=0x7f102c00d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:42.369 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.369+0000 7f1042ffd640 1 -- 172.21.15.83:0/1407112227 >> v1:172.21.15.83:6789/0 conn(0x7f10440a98f0 legacy=0x7f102c009a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:42.369 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.369+0000 7f1042ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:24:42.369 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.369+0000 7f1042ffd640 1 monclient: found mon.b 2024-03-25T19:24:42.369 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.369+0000 7f1042ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:42.369 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.369+0000 7f1042ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:24:42.369 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.369+0000 7f1042ffd640 1 -- 172.21.15.83:0/1407112227 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=2+}) v3 -- 0x7f102c011f50 con 0x7f102c003aa0 2024-03-25T19:24:42.369 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.369+0000 7f1042ffd640 10 monclient: _finish_auth 0 2024-03-25T19:24:42.370 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.369+0000 7f1042ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:24:42.370 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.369+0000 7f1042ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:42.370 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.369+0000 7f1042ffd640 1 -- 172.21.15.83:0/1407112227 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f1034001300 con 0x7f102c003aa0 2024-03-25T19:24:42.370 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.369+0000 7f1042ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:42.370 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.369+0000 7f1042ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:24:42.370 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.369+0000 7f1042ffd640 10 monclient: dump: 2024-03-25T19:24:42.370 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:24:42.370 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:24:42.370 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:42.370 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:42.370 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:42.370 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:24:42.370 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:24:42.371 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:24:42.371 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:24:42.371 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:42.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.369+0000 7f104a474640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:24:42.379 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.379+0000 7f1042ffd640 1 -- 172.21.15.83:0/1407112227 <== mon.1 v1:172.21.15.103:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1034001460 con 0x7f102c003aa0 2024-03-25T19:24:42.379 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.379+0000 7f1042ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:42.379 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.379+0000 7f104a474640 10 monclient: get_monmap_and_config success 2024-03-25T19:24:42.379 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.379+0000 7f104a474640 4 set_mon_vals no callback set 2024-03-25T19:24:42.380 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.380+0000 7f104a474640 10 monclient: shutdown 2024-03-25T19:24:42.380 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.380+0000 7f104a474640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:42.380 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.380+0000 7f104a474640 1 -- 172.21.15.83:0/1407112227 >> v1:172.21.15.103:6789/0 conn(0x7f102c003aa0 legacy=0x7f102c003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:42.381 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.380+0000 7f104a474640 1 -- 172.21.15.83:0/1407112227 shutdown_connections 2024-03-25T19:24:42.381 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.380+0000 7f104a474640 1 -- 172.21.15.83:0/1407112227 >> 172.21.15.83:0/1407112227 conn(0x7f10440b7ea0 msgr2=0x7f10440b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:42.381 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.380+0000 7f104a474640 1 -- 172.21.15.83:0/1407112227 shutdown_connections 2024-03-25T19:24:42.381 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.380+0000 7f104a474640 1 -- 172.21.15.83:0/1407112227 wait complete. 2024-03-25T19:24:42.382 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.381+0000 7f104a474640 10 monclient: build_initial_monmap 2024-03-25T19:24:42.382 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.381+0000 7f104a474640 10 monclient: monmap: 2024-03-25T19:24:42.382 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:24:42.382 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:24:42.382 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:24:42.382 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:24:42.382 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:42.382 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:24:42.382 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:24:42.382 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:24:42.382 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:24:42.382 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:42.382 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.381+0000 7f104a474640 1 librados: starting msgr at 2024-03-25T19:24:42.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.382+0000 7f104a474640 1 librados: starting objecter 2024-03-25T19:24:42.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.382+0000 7f104a474640 1 Processor -- start 2024-03-25T19:24:42.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.382+0000 7f104a474640 1 -- start start 2024-03-25T19:24:42.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.382+0000 7f104a474640 1 librados: setting wanted keys 2024-03-25T19:24:42.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.382+0000 7f104a474640 1 librados: calling monclient init 2024-03-25T19:24:42.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.382+0000 7f104a474640 10 monclient: init 2024-03-25T19:24:42.384 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.382+0000 7f104a474640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:42.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.382+0000 7f104a474640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:24:42.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.382+0000 7f104a474640 10 monclient(hunting): picked mon.noname-a con 0x7f102c003aa0 addr v1:172.21.15.83:6789/0 2024-03-25T19:24:42.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.383+0000 7f104a474640 10 monclient(hunting): picked mon.noname-b con 0x7f10440a98f0 addr v1:172.21.15.103:6789/0 2024-03-25T19:24:42.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.383+0000 7f104a474640 10 monclient(hunting): picked mon.noname-c con 0x7f10441430c0 addr v1:172.21.15.83:6790/0 2024-03-25T19:24:42.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.383+0000 7f104a474640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1044138e50 con 0x7f102c003aa0 2024-03-25T19:24:42.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.383+0000 7f104a474640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f104413be00 con 0x7f10440a98f0 2024-03-25T19:24:42.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.383+0000 7f104a474640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f104413bf90 con 0x7f10441430c0 2024-03-25T19:24:42.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.383+0000 7f104a474640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:42.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.383+0000 7f104a474640 10 monclient(hunting): authenticate will time out at 6339.654949s 2024-03-25T19:24:42.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.383+0000 7f1043fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f102c003aa0 0x7f104413b6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49524/0 (socket says 172.21.15.83:49524) 2024-03-25T19:24:42.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.383+0000 7f1043fff640 1 -- 172.21.15.83:0/2963720220 learned_addr learned my addr 172.21.15.83:0/2963720220 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:24:42.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.384+0000 7f1040ff9640 10 client.?.objecter ms_handle_connect 0x7f102c003aa0 2024-03-25T19:24:42.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.384+0000 7f1040ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:42.405 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.406+0000 7f1040ff9640 1 -- 172.21.15.83:0/2963720220 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3068338260 0 0) 0x7f1044138e50 con 0x7f102c003aa0 2024-03-25T19:24:42.405 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.406+0000 7f1040ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:24:42.405 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.406+0000 7f1040ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:42.405 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.406+0000 7f1040ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:42.406 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.406+0000 7f1040ff9640 10 monclient(hunting): my global_id is 4443 2024-03-25T19:24:42.406 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.406+0000 7f1040ff9640 1 -- 172.21.15.83:0/2963720220 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1030004eb0 con 0x7f102c003aa0 2024-03-25T19:24:42.408 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.409+0000 7f1040ff9640 10 client.?.objecter ms_handle_connect 0x7f10440a98f0 2024-03-25T19:24:42.409 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.409+0000 7f1040ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:42.413 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.413+0000 7f1040ff9640 10 client.?.objecter ms_handle_connect 0x7f10441430c0 2024-03-25T19:24:42.413 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.413+0000 7f1040ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:42.417 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.417+0000 7f1040ff9640 1 -- 172.21.15.83:0/2963720220 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2925154903 0 0) 0x7f1030004eb0 con 0x7f102c003aa0 2024-03-25T19:24:42.417 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.417+0000 7f1040ff9640 1 -- 172.21.15.83:0/2963720220 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1044138e50 con 0x7f102c003aa0 2024-03-25T19:24:42.417 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.418+0000 7f1040ff9640 1 -- 172.21.15.83:0/2963720220 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f1034001a70 con 0x7f102c003aa0 2024-03-25T19:24:42.417 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.418+0000 7f1040ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:42.418 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.418+0000 7f1040ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:24:42.418 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.418+0000 7f1040ff9640 10 monclient(hunting): dump: 2024-03-25T19:24:42.418 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:24:42.418 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:24:42.418 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:42.418 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:42.418 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:42.418 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:24:42.418 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:24:42.418 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:24:42.418 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:24:42.418 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:42.418 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.418+0000 7f1040ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:24:42.421 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.421+0000 7f1040ff9640 1 -- 172.21.15.83:0/2963720220 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 652479978 0 0) 0x7f104413bf90 con 0x7f10441430c0 2024-03-25T19:24:42.421 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.422+0000 7f1040ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:24:42.421 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.422+0000 7f1040ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:42.421 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.422+0000 7f1040ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:42.421 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.422+0000 7f1040ff9640 10 monclient(hunting): my global_id is 4460 2024-03-25T19:24:42.421 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.422+0000 7f1040ff9640 1 -- 172.21.15.83:0/2963720220 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1030004eb0 con 0x7f10441430c0 2024-03-25T19:24:42.431 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.431+0000 7f1040ff9640 1 -- 172.21.15.83:0/2963720220 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4068330715 0 0) 0x7f104413be00 con 0x7f10440a98f0 2024-03-25T19:24:42.431 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.431+0000 7f1040ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:24:42.431 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.431+0000 7f1040ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:42.432 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.431+0000 7f1040ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:42.432 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.431+0000 7f1040ff9640 10 monclient(hunting): my global_id is 4453 2024-03-25T19:24:42.432 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.431+0000 7f1040ff9640 1 -- 172.21.15.83:0/2963720220 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f104413bf90 con 0x7f10440a98f0 2024-03-25T19:24:42.432 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.431+0000 7f1040ff9640 1 -- 172.21.15.83:0/2963720220 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1768795614 0 0) 0x7f1044138e50 con 0x7f102c003aa0 2024-03-25T19:24:42.432 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.431+0000 7f1040ff9640 1 -- 172.21.15.83:0/2963720220 >> v1:172.21.15.83:6790/0 conn(0x7f10441430c0 legacy=0x7f1044138740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:42.432 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.431+0000 7f1040ff9640 1 -- 172.21.15.83:0/2963720220 >> v1:172.21.15.103:6789/0 conn(0x7f10440a98f0 legacy=0x7f1044138030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:42.432 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.431+0000 7f1040ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:24:42.432 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.431+0000 7f1040ff9640 1 monclient: found mon.a 2024-03-25T19:24:42.432 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.431+0000 7f1040ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:42.432 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.431+0000 7f1040ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:24:42.432 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.431+0000 7f1040ff9640 1 -- 172.21.15.83:0/2963720220 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10441c9e60 con 0x7f102c003aa0 2024-03-25T19:24:42.432 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.431+0000 7f1040ff9640 10 monclient: _finish_auth 0 2024-03-25T19:24:42.432 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.431+0000 7f1040ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:24:42.432 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.431+0000 7f1040ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:42.432 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.431+0000 7f104a474640 5 monclient: authenticate success, global_id 4443 2024-03-25T19:24:42.432 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.431+0000 7f104a474640 10 monclient: _renew_subs 2024-03-25T19:24:42.433 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.431+0000 7f104a474640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:24:42.433 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.431+0000 7f104a474640 1 -- 172.21.15.83:0/2963720220 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10441401f0 con 0x7f102c003aa0 2024-03-25T19:24:42.433 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.431+0000 7f104a474640 10 client.4443.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:24:42.433 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.431+0000 7f104a474640 10 monclient: _renew_subs 2024-03-25T19:24:42.433 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.431+0000 7f104a474640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:24:42.433 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.431+0000 7f104a474640 1 -- 172.21.15.83:0/2963720220 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f10441c9360 con 0x7f102c003aa0 2024-03-25T19:24:42.433 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.431+0000 7f104a474640 1 librados: init done 2024-03-25T19:24:42.433 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.432+0000 7f104a474640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:42.433 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.432+0000 7f104a474640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:42.434 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.432+0000 7f104a474640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:24:42.434 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.432+0000 7f104a474640 1 -- 172.21.15.83:0/2963720220 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f104413a2e0 con 0x7f102c003aa0 2024-03-25T19:24:42.452 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.452+0000 7f1040ff9640 1 -- 172.21.15.83:0/2963720220 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1034001bd0 con 0x7f102c003aa0 2024-03-25T19:24:42.452 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.452+0000 7f1040ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:42.452 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.452+0000 7f1040ff9640 1 -- 172.21.15.83:0/2963720220 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f1034003b90 con 0x7f102c003aa0 2024-03-25T19:24:42.452 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.452+0000 7f1040ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:42.453 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.452+0000 7f1040ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:24:42.453 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.452+0000 7f1040ff9640 10 monclient: dump: 2024-03-25T19:24:42.453 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:24:42.453 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:24:42.453 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:42.453 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:42.453 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:42.453 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:24:42.453 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:24:42.453 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:24:42.453 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:24:42.453 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:42.453 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.452+0000 7f10427fc640 4 set_mon_vals no callback set 2024-03-25T19:24:42.454 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.454+0000 7f1040ff9640 1 -- 172.21.15.83:0/2963720220 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7f1034003db0 con 0x7f102c003aa0 2024-03-25T19:24:42.454 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.454+0000 7f10437fe640 10 monclient: get_auth_request con 0x7f103006de00 auth_method 0 2024-03-25T19:24:42.454 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.454+0000 7f1040ff9640 10 client.4443.objecter ms_handle_connect 0x7f103006de00 2024-03-25T19:24:42.457 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.457+0000 7f1040ff9640 1 -- 172.21.15.83:0/2963720220 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7f1034054990 con 0x7f102c003aa0 2024-03-25T19:24:42.457 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.458+0000 7f1040ff9640 10 client.4443.objecter ms_dispatch 0x7f1044000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:24:42.457 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.458+0000 7f1040ff9640 3 client.4443.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:24:42.457 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.458+0000 7f1040ff9640 3 client.4443.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:24:42.457 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.458+0000 7f1040ff9640 20 client.4443.objecter dump_active .. 0 homeless 2024-03-25T19:24:42.777 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.777+0000 7f1040ff9640 1 -- 172.21.15.83:0/2963720220 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f10340504a0 con 0x7f102c003aa0 2024-03-25T19:24:42.777 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.777+0000 7f1040ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:42.777 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:42.777+0000 7f1040ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:24:42.984 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:05:24.912 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:43.018 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:43.019+0000 7f104a474640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:43.019 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:43.019+0000 7f104a474640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:43.019 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:43.019+0000 7f104a474640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:24:43.019 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:43.019+0000 7f104a474640 1 -- 172.21.15.83:0/2963720220 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f10440627b0 con 0x7f102c003aa0 2024-03-25T19:24:43.038 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:05:24.966 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:43.084 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:43.084+0000 7f1040ff9640 1 -- 172.21.15.83:0/2963720220 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7f10340523d0 con 0x7f102c003aa0 2024-03-25T19:24:43.084 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:43.084+0000 7f1040ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:43.084 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:43.084+0000 7f1040ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:24:43.085 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:24:43.085 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:24:43.088 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:43.087+0000 7f104a474640 10 librados: watch_flush enter 2024-03-25T19:24:43.088 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:43.087+0000 7f104a474640 10 librados: watch_flush exit 2024-03-25T19:24:43.088 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:43.087+0000 7f104a474640 20 client.4443.objecter shutdown clearing up homeless session... 2024-03-25T19:24:43.088 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:43.087+0000 7f104a474640 10 client.4443.objecter successfully canceled tick 2024-03-25T19:24:43.088 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:43.087+0000 7f104a474640 1 -- 172.21.15.83:0/2963720220 >> v1:172.21.15.103:6800/345072327 conn(0x7f103006de00 legacy=0x7f10300702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:43.088 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:43.087+0000 7f104a474640 10 monclient: shutdown 2024-03-25T19:24:43.088 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:43.087+0000 7f104a474640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:43.088 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:43.087+0000 7f104a474640 1 -- 172.21.15.83:0/2963720220 >> v1:172.21.15.83:6789/0 conn(0x7f102c003aa0 legacy=0x7f104413b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:43.088 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:43.088+0000 7f104a474640 1 -- 172.21.15.83:0/2963720220 shutdown_connections 2024-03-25T19:24:43.088 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:43.088+0000 7f104a474640 1 -- 172.21.15.83:0/2963720220 >> 172.21.15.83:0/2963720220 conn(0x7f10440b7ea0 msgr2=0x7f1044074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:43.088 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:43.088+0000 7f104a474640 1 -- 172.21.15.83:0/2963720220 shutdown_connections 2024-03-25T19:24:43.088 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:43.088+0000 7f104a474640 1 -- 172.21.15.83:0/2963720220 wait complete. 2024-03-25T19:24:43.088 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:43.088+0000 7f104a474640 1 librados: shutdown 2024-03-25T19:24:43.094 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:05:25.022 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:43.098 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:24:43.109 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:05:25.036 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:43.117 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:05:25.044 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:43.137 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:05:25.047 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:43.146 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:05:25.056 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:43.148 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:05:25.076 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:43.180 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:05:25.090 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:49.905 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:24:50.022 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.022+0000 7fb9f004e640 10 monclient: get_monmap_and_config 2024-03-25T19:24:50.022 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.022+0000 7fb9f004e640 10 monclient: build_initial_monmap 2024-03-25T19:24:50.022 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.022+0000 7fb9f004e640 10 monclient: monmap: 2024-03-25T19:24:50.022 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:24:50.022 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:24:50.022 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:24:50.023509+0000 2024-03-25T19:24:50.023 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:24:50.023509+0000 2024-03-25T19:24:50.023 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:50.023 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:24:50.023 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:24:50.023 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:24:50.023 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:24:50.023 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:50.024 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.023+0000 7fb9f004e640 1 Processor -- start 2024-03-25T19:24:50.024 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.023+0000 7fb9f004e640 1 -- start start 2024-03-25T19:24:50.024 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.023+0000 7fb9f004e640 10 monclient: init 2024-03-25T19:24:50.024 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.024+0000 7fb9f004e640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:50.024 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.024+0000 7fb9f004e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:24:50.024 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.024+0000 7fb9f004e640 10 monclient(hunting): picked mon.noname-a con 0x7fb9e80a98f0 addr v1:172.21.15.83:6789/0 2024-03-25T19:24:50.024 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.024+0000 7fb9f004e640 10 monclient(hunting): picked mon.noname-b con 0x7fb9e813f710 addr v1:172.21.15.83:6790/0 2024-03-25T19:24:50.025 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.024+0000 7fb9f004e640 10 monclient(hunting): picked mon.noname-c con 0x7fb9e81430c0 addr v1:172.21.15.103:6789/0 2024-03-25T19:24:50.025 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.024+0000 7fb9f004e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9e8142520 con 0x7fb9e80a98f0 2024-03-25T19:24:50.025 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.024+0000 7fb9f004e640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9e80b28a0 con 0x7fb9e81430c0 2024-03-25T19:24:50.025 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.024+0000 7fb9f004e640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9e80b2a30 con 0x7fb9e813f710 2024-03-25T19:24:50.025 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.024+0000 7fb9f004e640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:50.025 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.024+0000 7fb9f004e640 10 monclient(hunting): authenticate will time out at 6347.295984s 2024-03-25T19:24:50.031 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.031+0000 7fb9ee5eb640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fb9e81430c0 0x7fb9e814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:42870/0 (socket says 172.21.15.83:42870) 2024-03-25T19:24:50.031 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.031+0000 7fb9ee5eb640 1 -- 172.21.15.83:0/2931527298 learned_addr learned my addr 172.21.15.83:0/2931527298 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:24:50.041 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.042+0000 7fb9ecde8640 1 -- 172.21.15.83:0/2931527298 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3843209893 0 0) 0x7fb9e80b28a0 con 0x7fb9e81430c0 2024-03-25T19:24:50.042 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.042+0000 7fb9ecde8640 10 monclient(hunting): _negotiate 2024-03-25T19:24:50.042 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.042+0000 7fb9ecde8640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:50.042 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.042+0000 7fb9ecde8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:50.042 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.042+0000 7fb9ecde8640 10 monclient(hunting): my global_id is 4456 2024-03-25T19:24:50.042 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.042+0000 7fb9ecde8640 1 -- 172.21.15.83:0/2931527298 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9d4003520 con 0x7fb9e81430c0 2024-03-25T19:24:50.049 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.049+0000 7fb9ecde8640 1 -- 172.21.15.83:0/2931527298 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3765790122 0 0) 0x7fb9d4003520 con 0x7fb9e81430c0 2024-03-25T19:24:50.050 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.049+0000 7fb9ecde8640 1 -- 172.21.15.83:0/2931527298 >> v1:172.21.15.83:6790/0 conn(0x7fb9e813f710 legacy=0x7fb9e81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:50.050 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.049+0000 7fb9ecde8640 1 -- 172.21.15.83:0/2931527298 >> v1:172.21.15.83:6789/0 conn(0x7fb9e80a98f0 legacy=0x7fb9e80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:50.050 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.050+0000 7fb9ecde8640 10 monclient: _finish_hunting 0 2024-03-25T19:24:50.050 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.050+0000 7fb9ecde8640 1 monclient: found mon.noname-c 2024-03-25T19:24:50.050 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.050+0000 7fb9ecde8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:50.050 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.050+0000 7fb9ecde8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.103:6789/0 2024-03-25T19:24:50.050 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.050+0000 7fb9ecde8640 1 -- 172.21.15.83:0/2931527298 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9e80b44c0 con 0x7fb9e81430c0 2024-03-25T19:24:50.050 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.050+0000 7fb9ecde8640 10 monclient: _finish_auth 0 2024-03-25T19:24:50.050 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.050+0000 7fb9ecde8640 10 monclient: _check_auth_tickets 2024-03-25T19:24:50.050 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.050+0000 7fb9ecde8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:50.050 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.050+0000 7fb9ecde8640 1 -- 172.21.15.83:0/2931527298 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fb9e0001a10 con 0x7fb9e81430c0 2024-03-25T19:24:50.050 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.050+0000 7fb9f004e640 5 monclient: authenticate success, global_id 4456 2024-03-25T19:24:50.050 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.050+0000 7fb9f004e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:24:50.051 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.050+0000 7fb9ecde8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:50.051 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.050+0000 7fb9ecde8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:24:50.051 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.050+0000 7fb9ecde8640 10 monclient: dump: 2024-03-25T19:24:50.051 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:24:50.051 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:24:50.051 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:50.051 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:50.051 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:50.051 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:24:50.051 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:24:50.051 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:24:50.051 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:24:50.051 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:50.051 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.050+0000 7fb9f004e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:24:50.069 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.069+0000 7fb9ecde8640 1 -- 172.21.15.83:0/2931527298 <== mon.1 v1:172.21.15.103:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9e0001b70 con 0x7fb9e81430c0 2024-03-25T19:24:50.069 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.069+0000 7fb9ecde8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:50.069 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.069+0000 7fb9ecde8640 1 -- 172.21.15.83:0/2931527298 <== mon.1 v1:172.21.15.103:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fb9e0001f80 con 0x7fb9e81430c0 2024-03-25T19:24:50.069 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.069+0000 7fb9f004e640 10 monclient: get_monmap_and_config success 2024-03-25T19:24:50.070 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.069+0000 7fb9f004e640 4 set_mon_vals no callback set 2024-03-25T19:24:50.070 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.069+0000 7fb9ecde8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:50.070 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.069+0000 7fb9ecde8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:24:50.070 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.069+0000 7fb9ecde8640 10 monclient: dump: 2024-03-25T19:24:50.070 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:24:50.070 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:24:50.070 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:50.070 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:50.070 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:50.070 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:24:50.070 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:24:50.070 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:24:50.070 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:24:50.071 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:50.073 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.070+0000 7fb9f004e640 10 monclient: shutdown 2024-03-25T19:24:50.073 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.070+0000 7fb9f004e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:50.073 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.070+0000 7fb9f004e640 1 -- 172.21.15.83:0/2931527298 >> v1:172.21.15.103:6789/0 conn(0x7fb9e81430c0 legacy=0x7fb9e814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:50.073 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.070+0000 7fb9f004e640 1 -- 172.21.15.83:0/2931527298 shutdown_connections 2024-03-25T19:24:50.073 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.070+0000 7fb9f004e640 1 -- 172.21.15.83:0/2931527298 >> 172.21.15.83:0/2931527298 conn(0x7fb9e80b7ea0 msgr2=0x7fb9e80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:50.073 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.070+0000 7fb9f004e640 1 -- 172.21.15.83:0/2931527298 shutdown_connections 2024-03-25T19:24:50.073 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.070+0000 7fb9f004e640 1 -- 172.21.15.83:0/2931527298 wait complete. 2024-03-25T19:24:50.073 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.071+0000 7fb9f004e640 10 monclient: build_initial_monmap 2024-03-25T19:24:50.073 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.071+0000 7fb9f004e640 10 monclient: monmap: 2024-03-25T19:24:50.073 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:24:50.073 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:24:50.073 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:24:50.073 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:24:50.073 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:50.074 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:24:50.074 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:24:50.074 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:24:50.074 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:24:50.074 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:50.074 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.071+0000 7fb9f004e640 1 librados: starting msgr at 2024-03-25T19:24:50.074 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.071+0000 7fb9f004e640 1 librados: starting objecter 2024-03-25T19:24:50.074 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.071+0000 7fb9f004e640 1 Processor -- start 2024-03-25T19:24:50.074 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.071+0000 7fb9f004e640 1 -- start start 2024-03-25T19:24:50.074 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.071+0000 7fb9f004e640 1 librados: setting wanted keys 2024-03-25T19:24:50.074 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.071+0000 7fb9f004e640 1 librados: calling monclient init 2024-03-25T19:24:50.074 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.071+0000 7fb9f004e640 10 monclient: init 2024-03-25T19:24:50.074 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.072+0000 7fb9f004e640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:50.074 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.072+0000 7fb9f004e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:24:50.075 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.072+0000 7fb9f004e640 10 monclient(hunting): picked mon.noname-a con 0x7fb9e80a98f0 addr v1:172.21.15.83:6789/0 2024-03-25T19:24:50.075 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.072+0000 7fb9f004e640 10 monclient(hunting): picked mon.noname-b con 0x7fb9e813f710 addr v1:172.21.15.103:6789/0 2024-03-25T19:24:50.075 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.072+0000 7fb9f004e640 10 monclient(hunting): picked mon.noname-c con 0x7fb9e81430c0 addr v1:172.21.15.83:6790/0 2024-03-25T19:24:50.075 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.072+0000 7fb9f004e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9e81396e0 con 0x7fb9e80a98f0 2024-03-25T19:24:50.075 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.072+0000 7fb9f004e640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9e8138530 con 0x7fb9e813f710 2024-03-25T19:24:50.075 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.072+0000 7fb9f004e640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9e81399c0 con 0x7fb9e81430c0 2024-03-25T19:24:50.075 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.072+0000 7fb9f004e640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:50.075 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.072+0000 7fb9f004e640 10 monclient(hunting): authenticate will time out at 6347.344504s 2024-03-25T19:24:50.085 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.085+0000 7fb9ed5e9640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fb9e813f710 0x7fb9e81da850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:42918/0 (socket says 172.21.15.83:42918) 2024-03-25T19:24:50.085 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.085+0000 7fb9ed5e9640 1 -- 172.21.15.83:0/1203909611 learned_addr learned my addr 172.21.15.83:0/1203909611 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:24:50.085 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.085+0000 7fb9de7fc640 10 client.?.objecter ms_handle_connect 0x7fb9e813f710 2024-03-25T19:24:50.085 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.085+0000 7fb9de7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:50.091 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.092+0000 7fb9de7fc640 10 client.?.objecter ms_handle_connect 0x7fb9e80a98f0 2024-03-25T19:24:50.091 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.092+0000 7fb9de7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:50.093 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.094+0000 7fb9de7fc640 1 -- 172.21.15.83:0/1203909611 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2883183060 0 0) 0x7fb9e8138530 con 0x7fb9e813f710 2024-03-25T19:24:50.093 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.094+0000 7fb9de7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:24:50.094 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.094+0000 7fb9de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:50.094 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.094+0000 7fb9de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:50.094 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.094+0000 7fb9de7fc640 10 monclient(hunting): my global_id is 4459 2024-03-25T19:24:50.094 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.094+0000 7fb9de7fc640 1 -- 172.21.15.83:0/1203909611 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9cc004eb0 con 0x7fb9e813f710 2024-03-25T19:24:50.100 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.100+0000 7fb9de7fc640 10 client.?.objecter ms_handle_connect 0x7fb9e81430c0 2024-03-25T19:24:50.100 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.100+0000 7fb9de7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:50.105 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.105+0000 7fb9de7fc640 1 -- 172.21.15.83:0/1203909611 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4261893450 0 0) 0x7fb9e81396e0 con 0x7fb9e80a98f0 2024-03-25T19:24:50.105 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.105+0000 7fb9de7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:24:50.105 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.105+0000 7fb9de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:50.105 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.105+0000 7fb9de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:50.105 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.105+0000 7fb9de7fc640 10 monclient(hunting): my global_id is 4449 2024-03-25T19:24:50.105 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.105+0000 7fb9de7fc640 1 -- 172.21.15.83:0/1203909611 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9e8138530 con 0x7fb9e80a98f0 2024-03-25T19:24:50.117 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.117+0000 7fb9de7fc640 1 -- 172.21.15.83:0/1203909611 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2822890719 0 0) 0x7fb9cc004eb0 con 0x7fb9e813f710 2024-03-25T19:24:50.117 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.117+0000 7fb9de7fc640 1 -- 172.21.15.83:0/1203909611 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9e81396e0 con 0x7fb9e813f710 2024-03-25T19:24:50.117 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.117+0000 7fb9de7fc640 1 -- 172.21.15.83:0/1203909611 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fb9d8001a30 con 0x7fb9e813f710 2024-03-25T19:24:50.117 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.117+0000 7fb9de7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:50.117 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.117+0000 7fb9de7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:24:50.117 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.117+0000 7fb9de7fc640 10 monclient(hunting): dump: 2024-03-25T19:24:50.117 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:24:50.117 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:24:50.117 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:50.117 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:50.117 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:50.117 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:24:50.118 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:24:50.118 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:24:50.118 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:24:50.118 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:50.118 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.117+0000 7fb9de7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:24:50.119 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.120+0000 7fb9de7fc640 1 -- 172.21.15.83:0/1203909611 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 532197457 0 0) 0x7fb9e81399c0 con 0x7fb9e81430c0 2024-03-25T19:24:50.120 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.120+0000 7fb9de7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:24:50.120 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.120+0000 7fb9de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:50.120 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.120+0000 7fb9de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:50.120 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.120+0000 7fb9de7fc640 10 monclient(hunting): my global_id is 4466 2024-03-25T19:24:50.120 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.120+0000 7fb9de7fc640 1 -- 172.21.15.83:0/1203909611 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9cc004eb0 con 0x7fb9e81430c0 2024-03-25T19:24:50.121 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.121+0000 7fb9de7fc640 1 -- 172.21.15.83:0/1203909611 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1341341996 0 0) 0x7fb9e8138530 con 0x7fb9e80a98f0 2024-03-25T19:24:50.121 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.121+0000 7fb9de7fc640 1 -- 172.21.15.83:0/1203909611 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9e81399c0 con 0x7fb9e80a98f0 2024-03-25T19:24:50.121 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.121+0000 7fb9de7fc640 1 -- 172.21.15.83:0/1203909611 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fb9e40018a0 con 0x7fb9e80a98f0 2024-03-25T19:24:50.121 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.121+0000 7fb9de7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:50.121 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.121+0000 7fb9de7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:24:50.121 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.121+0000 7fb9de7fc640 10 monclient(hunting): dump: 2024-03-25T19:24:50.121 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:24:50.121 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:24:50.122 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:50.122 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:50.122 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:50.122 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:24:50.122 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:24:50.122 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:24:50.122 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:24:50.122 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:50.127 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.127+0000 7fb9de7fc640 1 -- 172.21.15.83:0/1203909611 <== mon.1 v1:172.21.15.103:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3288966508 0 0) 0x7fb9e81396e0 con 0x7fb9e813f710 2024-03-25T19:24:50.127 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.127+0000 7fb9de7fc640 1 -- 172.21.15.83:0/1203909611 >> v1:172.21.15.83:6790/0 conn(0x7fb9e81430c0 legacy=0x7fb9e81e1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:50.127 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.127+0000 7fb9de7fc640 1 -- 172.21.15.83:0/1203909611 >> v1:172.21.15.83:6789/0 conn(0x7fb9e80a98f0 legacy=0x7fb9e8137820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:50.127 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.127+0000 7fb9de7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:24:50.127 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.127+0000 7fb9de7fc640 1 monclient: found mon.b 2024-03-25T19:24:50.127 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.127+0000 7fb9de7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:50.128 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.127+0000 7fb9de7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:24:50.128 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.127+0000 7fb9de7fc640 1 -- 172.21.15.83:0/1203909611 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9e8138a50 con 0x7fb9e813f710 2024-03-25T19:24:50.128 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.127+0000 7fb9de7fc640 10 monclient: _finish_auth 0 2024-03-25T19:24:50.128 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.127+0000 7fb9de7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:24:50.128 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.127+0000 7fb9de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:50.128 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.127+0000 7fb9f004e640 5 monclient: authenticate success, global_id 4459 2024-03-25T19:24:50.128 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.127+0000 7fb9f004e640 10 monclient: _renew_subs 2024-03-25T19:24:50.128 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.127+0000 7fb9f004e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:24:50.128 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.127+0000 7fb9f004e640 1 -- 172.21.15.83:0/1203909611 --> v1:172.21.15.103:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9e81380a0 con 0x7fb9e813f710 2024-03-25T19:24:50.128 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.127+0000 7fb9f004e640 10 client.4459.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:24:50.128 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.127+0000 7fb9f004e640 10 monclient: _renew_subs 2024-03-25T19:24:50.128 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.127+0000 7fb9f004e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:24:50.128 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.127+0000 7fb9f004e640 1 -- 172.21.15.83:0/1203909611 --> v1:172.21.15.103:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9e81db140 con 0x7fb9e813f710 2024-03-25T19:24:50.128 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.127+0000 7fb9f004e640 1 librados: init done 2024-03-25T19:24:50.129 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.128+0000 7fb9f004e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:50.129 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.128+0000 7fb9f004e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:50.129 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.128+0000 7fb9f004e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:24:50.129 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.128+0000 7fb9f004e640 1 -- 172.21.15.83:0/1203909611 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9e8138ca0 con 0x7fb9e813f710 2024-03-25T19:24:50.423 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.424+0000 7fb9de7fc640 1 -- 172.21.15.83:0/1203909611 <== mon.1 v1:172.21.15.103:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9d8001c70 con 0x7fb9e813f710 2024-03-25T19:24:50.424 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.424+0000 7fb9de7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:50.424 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.424+0000 7fb9de7fc640 1 -- 172.21.15.83:0/1203909611 <== mon.1 v1:172.21.15.103:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fb9d80038c0 con 0x7fb9e813f710 2024-03-25T19:24:50.424 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.424+0000 7fb9de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:50.424 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.424+0000 7fb9de7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:24:50.424 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.424+0000 7fb9de7fc640 10 monclient: dump: 2024-03-25T19:24:50.424 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:24:50.424 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:24:50.424 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:50.424 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:50.424 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:50.424 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:24:50.424 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:24:50.424 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:24:50.424 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:24:50.425 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:50.425 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.424+0000 7fb9dffff640 4 set_mon_vals no callback set 2024-03-25T19:24:50.425 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.424+0000 7fb9de7fc640 1 -- 172.21.15.83:0/1203909611 <== mon.1 v1:172.21.15.103:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7fb9d8003aa0 con 0x7fb9e813f710 2024-03-25T19:24:50.425 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.425+0000 7fb9eddea640 10 monclient: get_auth_request con 0x7fb9cc06dfb0 auth_method 0 2024-03-25T19:24:50.425 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.425+0000 7fb9de7fc640 10 client.4459.objecter ms_handle_connect 0x7fb9cc06dfb0 2024-03-25T19:24:50.429 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.429+0000 7fb9de7fc640 1 -- 172.21.15.83:0/1203909611 <== mon.1 v1:172.21.15.103:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7fb9d8054710 con 0x7fb9e813f710 2024-03-25T19:24:50.429 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.429+0000 7fb9de7fc640 10 client.4459.objecter ms_dispatch 0x7fb9e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:24:50.429 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.429+0000 7fb9de7fc640 3 client.4459.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:24:50.429 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.429+0000 7fb9de7fc640 3 client.4459.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:24:50.429 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.429+0000 7fb9de7fc640 20 client.4459.objecter dump_active .. 0 homeless 2024-03-25T19:24:50.429 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.430+0000 7fb9de7fc640 1 -- 172.21.15.83:0/1203909611 <== mon.1 v1:172.21.15.103:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb9d8050220 con 0x7fb9e813f710 2024-03-25T19:24:50.430 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.430+0000 7fb9de7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:50.430 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.430+0000 7fb9de7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:24:50.664 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.664+0000 7fb9f004e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:50.664 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.664+0000 7fb9f004e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:50.664 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.664+0000 7fb9f004e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:24:50.664 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.664+0000 7fb9f004e640 1 -- 172.21.15.83:0/1203909611 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb9e80627b0 con 0x7fb9e813f710 2024-03-25T19:24:50.690 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.690+0000 7fb9de7fc640 1 -- 172.21.15.83:0/1203909611 <== mon.1 v1:172.21.15.103:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7fb9d8052150 con 0x7fb9e813f710 2024-03-25T19:24:50.690 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.690+0000 7fb9de7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:50.690 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.690+0000 7fb9de7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:24:50.690 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:24:50.690 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:24:50.692 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.692+0000 7fb9f004e640 10 librados: watch_flush enter 2024-03-25T19:24:50.692 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.692+0000 7fb9f004e640 10 librados: watch_flush exit 2024-03-25T19:24:50.692 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.692+0000 7fb9f004e640 20 client.4459.objecter shutdown clearing up homeless session... 2024-03-25T19:24:50.692 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.692+0000 7fb9f004e640 10 client.4459.objecter successfully canceled tick 2024-03-25T19:24:50.693 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.692+0000 7fb9f004e640 1 -- 172.21.15.83:0/1203909611 >> v1:172.21.15.103:6800/345072327 conn(0x7fb9cc06dfb0 legacy=0x7fb9cc070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:50.693 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.692+0000 7fb9f004e640 10 monclient: shutdown 2024-03-25T19:24:50.693 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.692+0000 7fb9f004e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:50.693 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.692+0000 7fb9f004e640 1 -- 172.21.15.83:0/1203909611 >> v1:172.21.15.103:6789/0 conn(0x7fb9e813f710 legacy=0x7fb9e81da850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:50.693 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.692+0000 7fb9f004e640 1 -- 172.21.15.83:0/1203909611 shutdown_connections 2024-03-25T19:24:50.693 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.692+0000 7fb9f004e640 1 -- 172.21.15.83:0/1203909611 >> 172.21.15.83:0/1203909611 conn(0x7fb9e80b7ea0 msgr2=0x7fb9e8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:50.693 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.692+0000 7fb9f004e640 1 -- 172.21.15.83:0/1203909611 shutdown_connections 2024-03-25T19:24:50.693 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.692+0000 7fb9f004e640 1 -- 172.21.15.83:0/1203909611 wait complete. 2024-03-25T19:24:50.693 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:50.692+0000 7fb9f004e640 1 librados: shutdown 2024-03-25T19:24:50.705 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:24:57.511 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:24:57.633 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.632+0000 7fc9b65ba640 10 monclient: get_monmap_and_config 2024-03-25T19:24:57.633 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.632+0000 7fc9b65ba640 10 monclient: build_initial_monmap 2024-03-25T19:24:57.633 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.632+0000 7fc9b65ba640 10 monclient: monmap: 2024-03-25T19:24:57.634 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:24:57.634 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:24:57.634 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:24:57.634761+0000 2024-03-25T19:24:57.634 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:24:57.634761+0000 2024-03-25T19:24:57.634 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:57.634 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:24:57.634 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:24:57.634 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:24:57.634 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:24:57.634 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:57.638 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.633+0000 7fc9b65ba640 1 Processor -- start 2024-03-25T19:24:57.638 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.633+0000 7fc9b65ba640 1 -- start start 2024-03-25T19:24:57.638 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.633+0000 7fc9b65ba640 10 monclient: init 2024-03-25T19:24:57.638 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.634+0000 7fc9b65ba640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:57.638 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.634+0000 7fc9b65ba640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:24:57.638 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.634+0000 7fc9b65ba640 10 monclient(hunting): picked mon.noname-a con 0x7fc9b00a98f0 addr v1:172.21.15.83:6789/0 2024-03-25T19:24:57.639 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.634+0000 7fc9b65ba640 10 monclient(hunting): picked mon.noname-c con 0x7fc9b013f710 addr v1:172.21.15.103:6789/0 2024-03-25T19:24:57.639 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.634+0000 7fc9b65ba640 10 monclient(hunting): picked mon.noname-b con 0x7fc9b01430c0 addr v1:172.21.15.83:6790/0 2024-03-25T19:24:57.639 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.634+0000 7fc9b65ba640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9b0142520 con 0x7fc9b00a98f0 2024-03-25T19:24:57.639 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.634+0000 7fc9b65ba640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9b00b28a0 con 0x7fc9b013f710 2024-03-25T19:24:57.639 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.634+0000 7fc9b65ba640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9b00b2a30 con 0x7fc9b01430c0 2024-03-25T19:24:57.639 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.634+0000 7fc9b65ba640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:57.639 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.635+0000 7fc9b65ba640 10 monclient(hunting): authenticate will time out at 6354.907748s 2024-03-25T19:24:57.639 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.638+0000 7fc9b4b57640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fc9b01430c0 0x7fc9b014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:59478/0 (socket says 172.21.15.83:59478) 2024-03-25T19:24:57.639 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.638+0000 7fc9b4b57640 1 -- 172.21.15.83:0/158134483 learned_addr learned my addr 172.21.15.83:0/158134483 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:24:57.649 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.648+0000 7fc9aeffd640 1 -- 172.21.15.83:0/158134483 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4030846243 0 0) 0x7fc9b00b2a30 con 0x7fc9b01430c0 2024-03-25T19:24:57.649 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.648+0000 7fc9aeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:24:57.649 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.648+0000 7fc9aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:57.649 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.648+0000 7fc9aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:57.649 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.648+0000 7fc9aeffd640 10 monclient(hunting): my global_id is 4469 2024-03-25T19:24:57.649 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.648+0000 7fc9aeffd640 1 -- 172.21.15.83:0/158134483 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc998003520 con 0x7fc9b01430c0 2024-03-25T19:24:57.666 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.665+0000 7fc9aeffd640 1 -- 172.21.15.83:0/158134483 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4080728907 0 0) 0x7fc9b0142520 con 0x7fc9b00a98f0 2024-03-25T19:24:57.666 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.665+0000 7fc9aeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:24:57.666 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.665+0000 7fc9aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:57.666 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.665+0000 7fc9aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:57.666 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.665+0000 7fc9aeffd640 10 monclient(hunting): my global_id is 4452 2024-03-25T19:24:57.666 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.665+0000 7fc9aeffd640 1 -- 172.21.15.83:0/158134483 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9b00b2a30 con 0x7fc9b00a98f0 2024-03-25T19:24:57.668 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.667+0000 7fc9aeffd640 1 -- 172.21.15.83:0/158134483 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1440121442 0 0) 0x7fc998003520 con 0x7fc9b01430c0 2024-03-25T19:24:57.668 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.667+0000 7fc9aeffd640 1 -- 172.21.15.83:0/158134483 >> v1:172.21.15.103:6789/0 conn(0x7fc9b013f710 legacy=0x7fc9b01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:57.668 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.667+0000 7fc9aeffd640 1 -- 172.21.15.83:0/158134483 >> v1:172.21.15.83:6789/0 conn(0x7fc9b00a98f0 legacy=0x7fc9b00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:57.668 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.667+0000 7fc9aeffd640 10 monclient: _finish_hunting 0 2024-03-25T19:24:57.668 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.667+0000 7fc9aeffd640 1 monclient: found mon.noname-b 2024-03-25T19:24:57.668 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.667+0000 7fc9aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:57.668 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.667+0000 7fc9aeffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.83:6790/0 2024-03-25T19:24:57.668 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.667+0000 7fc9aeffd640 1 -- 172.21.15.83:0/158134483 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9b00b44c0 con 0x7fc9b01430c0 2024-03-25T19:24:57.668 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.667+0000 7fc9aeffd640 10 monclient: _finish_auth 0 2024-03-25T19:24:57.668 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.667+0000 7fc9aeffd640 10 monclient: _check_auth_tickets 2024-03-25T19:24:57.669 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.667+0000 7fc9aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:57.669 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.667+0000 7fc9aeffd640 1 -- 172.21.15.83:0/158134483 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fc9a8001a10 con 0x7fc9b01430c0 2024-03-25T19:24:57.669 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.667+0000 7fc9aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:57.669 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.667+0000 7fc9aeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:24:57.669 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.667+0000 7fc9aeffd640 10 monclient: dump: 2024-03-25T19:24:57.669 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:24:57.669 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:24:57.669 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:57.669 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:57.669 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:57.669 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:24:57.669 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:24:57.669 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:24:57.669 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:24:57.670 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:57.670 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.667+0000 7fc9b65ba640 5 monclient: authenticate success, global_id 4469 2024-03-25T19:24:57.670 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.667+0000 7fc9b65ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:24:57.681 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.680+0000 7fc9aeffd640 1 -- 172.21.15.83:0/158134483 <== mon.2 v1:172.21.15.83:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9a8001b70 con 0x7fc9b01430c0 2024-03-25T19:24:57.681 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.680+0000 7fc9aeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:57.681 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.680+0000 7fc9aeffd640 1 -- 172.21.15.83:0/158134483 <== mon.2 v1:172.21.15.83:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fc9a8001f80 con 0x7fc9b01430c0 2024-03-25T19:24:57.681 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.680+0000 7fc9b65ba640 10 monclient: get_monmap_and_config success 2024-03-25T19:24:57.681 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.680+0000 7fc9b65ba640 4 set_mon_vals no callback set 2024-03-25T19:24:57.681 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.680+0000 7fc9aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:57.681 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.680+0000 7fc9aeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:24:57.682 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.680+0000 7fc9aeffd640 10 monclient: dump: 2024-03-25T19:24:57.682 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:24:57.682 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:24:57.682 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:57.682 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:57.682 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:57.682 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:24:57.682 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:24:57.682 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:24:57.682 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:24:57.682 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:57.684 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.681+0000 7fc9b65ba640 10 monclient: shutdown 2024-03-25T19:24:57.684 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.681+0000 7fc9b65ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:57.684 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.681+0000 7fc9b65ba640 1 -- 172.21.15.83:0/158134483 >> v1:172.21.15.83:6790/0 conn(0x7fc9b01430c0 legacy=0x7fc9b014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:57.684 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.681+0000 7fc9b65ba640 1 -- 172.21.15.83:0/158134483 shutdown_connections 2024-03-25T19:24:57.684 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.681+0000 7fc9b65ba640 1 -- 172.21.15.83:0/158134483 >> 172.21.15.83:0/158134483 conn(0x7fc9b00b7ea0 msgr2=0x7fc9b00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:57.684 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.681+0000 7fc9b65ba640 1 -- 172.21.15.83:0/158134483 shutdown_connections 2024-03-25T19:24:57.684 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.681+0000 7fc9b65ba640 1 -- 172.21.15.83:0/158134483 wait complete. 2024-03-25T19:24:57.684 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.682+0000 7fc9b65ba640 10 monclient: build_initial_monmap 2024-03-25T19:24:57.684 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.682+0000 7fc9b65ba640 10 monclient: monmap: 2024-03-25T19:24:57.684 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:24:57.684 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:24:57.685 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:24:57.685 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:24:57.685 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:57.685 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:24:57.685 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:24:57.685 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:24:57.685 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:24:57.685 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:57.685 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.683+0000 7fc9b65ba640 1 librados: starting msgr at 2024-03-25T19:24:57.685 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.683+0000 7fc9b65ba640 1 librados: starting objecter 2024-03-25T19:24:57.685 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.683+0000 7fc9b65ba640 1 Processor -- start 2024-03-25T19:24:57.685 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.683+0000 7fc9b65ba640 1 -- start start 2024-03-25T19:24:57.685 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.683+0000 7fc9b65ba640 1 librados: setting wanted keys 2024-03-25T19:24:57.685 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.683+0000 7fc9b65ba640 1 librados: calling monclient init 2024-03-25T19:24:57.685 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.683+0000 7fc9b65ba640 10 monclient: init 2024-03-25T19:24:57.686 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.684+0000 7fc9b65ba640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:57.686 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.684+0000 7fc9b65ba640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:24:57.687 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.684+0000 7fc9b65ba640 10 monclient(hunting): picked mon.noname-c con 0x7fc9b00a98f0 addr v1:172.21.15.83:6790/0 2024-03-25T19:24:57.687 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.684+0000 7fc9b65ba640 10 monclient(hunting): picked mon.noname-a con 0x7fc9b013f710 addr v1:172.21.15.83:6789/0 2024-03-25T19:24:57.687 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.684+0000 7fc9b65ba640 10 monclient(hunting): picked mon.noname-b con 0x7fc9b01430c0 addr v1:172.21.15.103:6789/0 2024-03-25T19:24:57.687 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.684+0000 7fc9b65ba640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9b013c2b0 con 0x7fc9b013f710 2024-03-25T19:24:57.687 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.684+0000 7fc9b65ba640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9b0136f10 con 0x7fc9b01430c0 2024-03-25T19:24:57.687 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.684+0000 7fc9b65ba640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9b01370a0 con 0x7fc9b00a98f0 2024-03-25T19:24:57.687 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.684+0000 7fc9b65ba640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:57.687 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.684+0000 7fc9b65ba640 10 monclient(hunting): authenticate will time out at 6354.956952s 2024-03-25T19:24:57.689 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.689+0000 7fc9af7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fc9b013f710 0x7fc9b01377d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:51248/0 (socket says 172.21.15.83:51248) 2024-03-25T19:24:57.690 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.689+0000 7fc9af7fe640 1 -- 172.21.15.83:0/952898779 learned_addr learned my addr 172.21.15.83:0/952898779 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:24:57.690 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.690+0000 7fc9acff9640 10 client.?.objecter ms_handle_connect 0x7fc9b013f710 2024-03-25T19:24:57.690 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.690+0000 7fc9acff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:57.692 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.691+0000 7fc9acff9640 10 client.?.objecter ms_handle_connect 0x7fc9b00a98f0 2024-03-25T19:24:57.692 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.691+0000 7fc9acff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:57.695 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.695+0000 7fc9acff9640 10 client.?.objecter ms_handle_connect 0x7fc9b01430c0 2024-03-25T19:24:57.695 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.695+0000 7fc9acff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:57.700 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.699+0000 7fc9acff9640 1 -- 172.21.15.83:0/952898779 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3801884337 0 0) 0x7fc9b0136f10 con 0x7fc9b01430c0 2024-03-25T19:24:57.700 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.699+0000 7fc9acff9640 10 monclient(hunting): _negotiate 2024-03-25T19:24:57.700 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.699+0000 7fc9acff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:57.700 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.699+0000 7fc9acff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:57.700 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.699+0000 7fc9acff9640 10 monclient(hunting): my global_id is 4465 2024-03-25T19:24:57.700 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.699+0000 7fc9acff9640 1 -- 172.21.15.83:0/952898779 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc99c004eb0 con 0x7fc9b01430c0 2024-03-25T19:24:57.704 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.703+0000 7fc9acff9640 1 -- 172.21.15.83:0/952898779 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2839966845 0 0) 0x7fc9b01370a0 con 0x7fc9b00a98f0 2024-03-25T19:24:57.704 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.703+0000 7fc9acff9640 10 monclient(hunting): _negotiate 2024-03-25T19:24:57.704 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.703+0000 7fc9acff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:57.704 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.703+0000 7fc9acff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:57.704 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.703+0000 7fc9acff9640 10 monclient(hunting): my global_id is 4472 2024-03-25T19:24:57.704 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.703+0000 7fc9acff9640 1 -- 172.21.15.83:0/952898779 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9b0136f10 con 0x7fc9b00a98f0 2024-03-25T19:24:57.707 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.706+0000 7fc9acff9640 1 -- 172.21.15.83:0/952898779 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3192381806 0 0) 0x7fc9b013c2b0 con 0x7fc9b013f710 2024-03-25T19:24:57.707 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.706+0000 7fc9acff9640 10 monclient(hunting): _negotiate 2024-03-25T19:24:57.707 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.706+0000 7fc9acff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:57.707 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.706+0000 7fc9acff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:57.707 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.706+0000 7fc9acff9640 10 monclient(hunting): my global_id is 4455 2024-03-25T19:24:57.707 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.706+0000 7fc9acff9640 1 -- 172.21.15.83:0/952898779 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9b01370a0 con 0x7fc9b013f710 2024-03-25T19:24:57.709 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.708+0000 7fc9acff9640 1 -- 172.21.15.83:0/952898779 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3641127474 0 0) 0x7fc99c004eb0 con 0x7fc9b01430c0 2024-03-25T19:24:57.709 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.708+0000 7fc9acff9640 1 -- 172.21.15.83:0/952898779 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc9b013c2b0 con 0x7fc9b01430c0 2024-03-25T19:24:57.709 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.708+0000 7fc9acff9640 1 -- 172.21.15.83:0/952898779 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fc9a8001e80 con 0x7fc9b01430c0 2024-03-25T19:24:57.709 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.708+0000 7fc9acff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:57.709 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.708+0000 7fc9acff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:24:57.709 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.708+0000 7fc9acff9640 10 monclient(hunting): dump: 2024-03-25T19:24:57.709 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:24:57.709 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:24:57.709 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:57.709 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:57.709 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:57.709 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:24:57.710 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:24:57.710 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:24:57.710 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:24:57.710 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:57.710 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.708+0000 7fc9acff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:24:57.718 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.716+0000 7fc9acff9640 1 -- 172.21.15.83:0/952898779 <== mon.1 v1:172.21.15.103:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1435217961 0 0) 0x7fc9b013c2b0 con 0x7fc9b01430c0 2024-03-25T19:24:57.718 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.716+0000 7fc9acff9640 1 -- 172.21.15.83:0/952898779 >> v1:172.21.15.83:6790/0 conn(0x7fc9b00a98f0 legacy=0x7fc9b01d7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:57.718 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.716+0000 7fc9acff9640 1 -- 172.21.15.83:0/952898779 >> v1:172.21.15.83:6789/0 conn(0x7fc9b013f710 legacy=0x7fc9b01377d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:57.718 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.716+0000 7fc9acff9640 10 monclient: _finish_hunting 0 2024-03-25T19:24:57.718 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.716+0000 7fc9acff9640 1 monclient: found mon.b 2024-03-25T19:24:57.718 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.716+0000 7fc9acff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:57.718 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.716+0000 7fc9acff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:24:57.718 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.716+0000 7fc9acff9640 1 -- 172.21.15.83:0/952898779 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9b0137230 con 0x7fc9b01430c0 2024-03-25T19:24:57.718 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.716+0000 7fc9acff9640 10 monclient: _finish_auth 0 2024-03-25T19:24:57.718 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.716+0000 7fc9acff9640 10 monclient: _check_auth_tickets 2024-03-25T19:24:57.718 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.716+0000 7fc9acff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:57.718 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.716+0000 7fc9b65ba640 5 monclient: authenticate success, global_id 4465 2024-03-25T19:24:57.718 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.716+0000 7fc9b65ba640 10 monclient: _renew_subs 2024-03-25T19:24:57.718 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.716+0000 7fc9b65ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:24:57.719 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.716+0000 7fc9b65ba640 1 -- 172.21.15.83:0/952898779 --> v1:172.21.15.103:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9b013b620 con 0x7fc9b01430c0 2024-03-25T19:24:57.719 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.717+0000 7fc9b65ba640 10 client.4465.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:24:57.719 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.717+0000 7fc9b65ba640 10 monclient: _renew_subs 2024-03-25T19:24:57.719 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.717+0000 7fc9b65ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:24:57.719 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.717+0000 7fc9b65ba640 1 -- 172.21.15.83:0/952898779 --> v1:172.21.15.103:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9b01338f0 con 0x7fc9b01430c0 2024-03-25T19:24:57.719 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.717+0000 7fc9b65ba640 1 librados: init done 2024-03-25T19:24:57.719 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.718+0000 7fc9b65ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:57.720 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.718+0000 7fc9b65ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:57.720 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.718+0000 7fc9b65ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:24:57.720 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:57.718+0000 7fc9b65ba640 1 -- 172.21.15.83:0/952898779 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9b013b9c0 con 0x7fc9b01430c0 2024-03-25T19:24:58.023 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:58.022+0000 7fc9acff9640 1 -- 172.21.15.83:0/952898779 <== mon.1 v1:172.21.15.103:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9a8002020 con 0x7fc9b01430c0 2024-03-25T19:24:58.023 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:58.022+0000 7fc9acff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:58.023 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:58.022+0000 7fc9acff9640 1 -- 172.21.15.83:0/952898779 <== mon.1 v1:172.21.15.103:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fc9a8004e10 con 0x7fc9b01430c0 2024-03-25T19:24:58.023 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:58.022+0000 7fc9acff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:58.023 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:58.022+0000 7fc9acff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:24:58.023 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:58.022+0000 7fc9acff9640 10 monclient: dump: 2024-03-25T19:24:58.023 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:24:58.024 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:24:58.024 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:58.024 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:24:58.024 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:58.024 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:24:58.024 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:24:58.024 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:24:58.024 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:24:58.024 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:24:58.024 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:58.022+0000 7fc9ae7fc640 4 set_mon_vals no callback set 2024-03-25T19:24:58.025 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:58.023+0000 7fc9acff9640 1 -- 172.21.15.83:0/952898779 <== mon.1 v1:172.21.15.103:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7fc9a801ade0 con 0x7fc9b01430c0 2024-03-25T19:24:58.025 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:58.024+0000 7fc9affff640 10 monclient: get_auth_request con 0x7fc99c06ddf0 auth_method 0 2024-03-25T19:24:58.025 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:58.024+0000 7fc9acff9640 10 client.4465.objecter ms_handle_connect 0x7fc99c06ddf0 2024-03-25T19:24:58.029 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:58.028+0000 7fc9acff9640 1 -- 172.21.15.83:0/952898779 <== mon.1 v1:172.21.15.103:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7fc9a8054900 con 0x7fc9b01430c0 2024-03-25T19:24:58.029 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:58.028+0000 7fc9acff9640 10 client.4465.objecter ms_dispatch 0x7fc9b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:24:58.029 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:58.028+0000 7fc9acff9640 3 client.4465.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:24:58.029 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:58.028+0000 7fc9acff9640 3 client.4465.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:24:58.029 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:58.028+0000 7fc9acff9640 20 client.4465.objecter dump_active .. 0 homeless 2024-03-25T19:24:58.030 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:58.029+0000 7fc9acff9640 1 -- 172.21.15.83:0/952898779 <== mon.1 v1:172.21.15.103:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc9a8050410 con 0x7fc9b01430c0 2024-03-25T19:24:58.030 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:58.029+0000 7fc9acff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:58.030 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:58.029+0000 7fc9acff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:24:58.273 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:58.272+0000 7fc9b65ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:58.273 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:58.272+0000 7fc9b65ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:58.273 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:58.272+0000 7fc9b65ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:24:58.273 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:58.272+0000 7fc9b65ba640 1 -- 172.21.15.83:0/952898779 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9b00627b0 con 0x7fc9b01430c0 2024-03-25T19:24:58.294 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:58.293+0000 7fc9acff9640 1 -- 172.21.15.83:0/952898779 <== mon.1 v1:172.21.15.103:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7fc9a8052340 con 0x7fc9b01430c0 2024-03-25T19:24:58.294 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:58.293+0000 7fc9acff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:58.294 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:58.293+0000 7fc9acff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:24:58.294 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:24:58.294 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:24:58.297 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:58.295+0000 7fc9b65ba640 10 librados: watch_flush enter 2024-03-25T19:24:58.297 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:58.295+0000 7fc9b65ba640 10 librados: watch_flush exit 2024-03-25T19:24:58.297 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:58.295+0000 7fc9b65ba640 20 client.4465.objecter shutdown clearing up homeless session... 2024-03-25T19:24:58.297 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:58.295+0000 7fc9b65ba640 10 client.4465.objecter successfully canceled tick 2024-03-25T19:24:58.297 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:58.295+0000 7fc9b65ba640 1 -- 172.21.15.83:0/952898779 >> v1:172.21.15.103:6800/345072327 conn(0x7fc99c06ddf0 legacy=0x7fc99c0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:58.297 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:58.295+0000 7fc9b65ba640 10 monclient: shutdown 2024-03-25T19:24:58.297 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:58.295+0000 7fc9b65ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:58.297 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:58.295+0000 7fc9b65ba640 1 -- 172.21.15.83:0/952898779 >> v1:172.21.15.103:6789/0 conn(0x7fc9b01430c0 legacy=0x7fc9b0133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:58.297 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:58.295+0000 7fc9b65ba640 1 -- 172.21.15.83:0/952898779 shutdown_connections 2024-03-25T19:24:58.297 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:58.295+0000 7fc9b65ba640 1 -- 172.21.15.83:0/952898779 >> 172.21.15.83:0/952898779 conn(0x7fc9b00b7ea0 msgr2=0x7fc9b0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:58.297 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:58.296+0000 7fc9b65ba640 1 -- 172.21.15.83:0/952898779 shutdown_connections 2024-03-25T19:24:58.297 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:58.296+0000 7fc9b65ba640 1 -- 172.21.15.83:0/952898779 wait complete. 2024-03-25T19:24:58.298 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:24:58.296+0000 7fc9b65ba640 1 librados: shutdown 2024-03-25T19:24:58.307 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:25:03.046 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:05:44.973 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:03.096 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:05:45.023 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:03.126 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:05:45.054 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:03.148 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:05:45.076 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:03.156 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:05:45.083 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:03.168 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:05:45.078 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:03.171 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:05:45.081 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:03.179 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:05:45.107 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:03.189 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:05:45.099 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:05.114 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:25:05.228 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.228+0000 7fc5d100e640 10 monclient: get_monmap_and_config 2024-03-25T19:25:05.228 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.228+0000 7fc5d100e640 10 monclient: build_initial_monmap 2024-03-25T19:25:05.229 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.228+0000 7fc5d100e640 10 monclient: monmap: 2024-03-25T19:25:05.229 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:25:05.229 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:25:05.229 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:25:05.229713+0000 2024-03-25T19:25:05.229 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:25:05.229713+0000 2024-03-25T19:25:05.229 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:05.229 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:25:05.229 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:25:05.229 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:25:05.229 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:25:05.229 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:05.231 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.229+0000 7fc5d100e640 1 Processor -- start 2024-03-25T19:25:05.231 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.229+0000 7fc5d100e640 1 -- start start 2024-03-25T19:25:05.231 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.229+0000 7fc5d100e640 10 monclient: init 2024-03-25T19:25:05.231 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.230+0000 7fc5d100e640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:05.231 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.230+0000 7fc5d100e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:25:05.231 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.230+0000 7fc5d100e640 10 monclient(hunting): picked mon.noname-c con 0x7fc5cc0a98f0 addr v1:172.21.15.103:6789/0 2024-03-25T19:25:05.231 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.230+0000 7fc5d100e640 10 monclient(hunting): picked mon.noname-a con 0x7fc5cc13f710 addr v1:172.21.15.83:6789/0 2024-03-25T19:25:05.231 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.230+0000 7fc5d100e640 10 monclient(hunting): picked mon.noname-b con 0x7fc5cc1430c0 addr v1:172.21.15.83:6790/0 2024-03-25T19:25:05.231 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.230+0000 7fc5d100e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5cc142520 con 0x7fc5cc13f710 2024-03-25T19:25:05.232 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.230+0000 7fc5d100e640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5cc0b28a0 con 0x7fc5cc0a98f0 2024-03-25T19:25:05.232 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.230+0000 7fc5d100e640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5cc0b2a30 con 0x7fc5cc1430c0 2024-03-25T19:25:05.232 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.230+0000 7fc5d100e640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:05.232 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.230+0000 7fc5d100e640 10 monclient(hunting): authenticate will time out at 6362.502314s 2024-03-25T19:25:05.240 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.240+0000 7fc5cad9d640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fc5cc0a98f0 0x7fc5cc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:42506/0 (socket says 172.21.15.83:42506) 2024-03-25T19:25:05.240 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.240+0000 7fc5cad9d640 1 -- 172.21.15.83:0/2273379939 learned_addr learned my addr 172.21.15.83:0/2273379939 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:25:05.255 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.255+0000 7fc5c9d9b640 1 -- 172.21.15.83:0/2273379939 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 621207439 0 0) 0x7fc5cc0b28a0 con 0x7fc5cc0a98f0 2024-03-25T19:25:05.255 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.255+0000 7fc5c9d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:25:05.255 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.255+0000 7fc5c9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:05.255 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.255+0000 7fc5c9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:05.255 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.255+0000 7fc5c9d9b640 10 monclient(hunting): my global_id is 4468 2024-03-25T19:25:05.255 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.255+0000 7fc5c9d9b640 1 -- 172.21.15.83:0/2273379939 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5b4003520 con 0x7fc5cc0a98f0 2024-03-25T19:25:05.262 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.263+0000 7fc5c9d9b640 1 -- 172.21.15.83:0/2273379939 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2166272344 0 0) 0x7fc5cc0b2a30 con 0x7fc5cc1430c0 2024-03-25T19:25:05.262 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.263+0000 7fc5c9d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:25:05.263 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.263+0000 7fc5c9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:05.263 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.263+0000 7fc5c9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:05.263 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.263+0000 7fc5c9d9b640 10 monclient(hunting): my global_id is 4475 2024-03-25T19:25:05.263 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.263+0000 7fc5c9d9b640 1 -- 172.21.15.83:0/2273379939 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5cc0b28a0 con 0x7fc5cc1430c0 2024-03-25T19:25:05.275 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.275+0000 7fc5c9d9b640 1 -- 172.21.15.83:0/2273379939 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2723863457 0 0) 0x7fc5cc142520 con 0x7fc5cc13f710 2024-03-25T19:25:05.275 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.275+0000 7fc5c9d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:25:05.275 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.275+0000 7fc5c9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:05.275 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.275+0000 7fc5c9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:05.275 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.275+0000 7fc5c9d9b640 10 monclient(hunting): my global_id is 4458 2024-03-25T19:25:05.275 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.275+0000 7fc5c9d9b640 1 -- 172.21.15.83:0/2273379939 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5cc0b2a30 con 0x7fc5cc13f710 2024-03-25T19:25:05.277 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.277+0000 7fc5c9d9b640 1 -- 172.21.15.83:0/2273379939 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 179397638 0 0) 0x7fc5b4003520 con 0x7fc5cc0a98f0 2024-03-25T19:25:05.277 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.277+0000 7fc5c9d9b640 1 -- 172.21.15.83:0/2273379939 >> v1:172.21.15.83:6790/0 conn(0x7fc5cc1430c0 legacy=0x7fc5cc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:05.277 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.277+0000 7fc5c9d9b640 1 -- 172.21.15.83:0/2273379939 >> v1:172.21.15.83:6789/0 conn(0x7fc5cc13f710 legacy=0x7fc5cc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:05.277 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.277+0000 7fc5c9d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:25:05.277 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.277+0000 7fc5c9d9b640 1 monclient: found mon.noname-c 2024-03-25T19:25:05.277 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.277+0000 7fc5c9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:05.277 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.277+0000 7fc5c9d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.103:6789/0 2024-03-25T19:25:05.277 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.277+0000 7fc5c9d9b640 1 -- 172.21.15.83:0/2273379939 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5cc0b44c0 con 0x7fc5cc0a98f0 2024-03-25T19:25:05.277 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.277+0000 7fc5c9d9b640 10 monclient: _finish_auth 0 2024-03-25T19:25:05.278 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.277+0000 7fc5c9d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:25:05.278 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.277+0000 7fc5c9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:05.278 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.277+0000 7fc5c9d9b640 1 -- 172.21.15.83:0/2273379939 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fc5bc001690 con 0x7fc5cc0a98f0 2024-03-25T19:25:05.278 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.277+0000 7fc5c9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:05.278 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.277+0000 7fc5c9d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:25:05.278 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.277+0000 7fc5c9d9b640 10 monclient: dump: 2024-03-25T19:25:05.278 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:25:05.278 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:25:05.278 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:05.278 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:05.278 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:05.278 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:25:05.278 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:25:05.279 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:25:05.279 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:25:05.279 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:05.279 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.277+0000 7fc5d100e640 5 monclient: authenticate success, global_id 4468 2024-03-25T19:25:05.279 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.277+0000 7fc5d100e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:25:05.295 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.296+0000 7fc5c9d9b640 1 -- 172.21.15.83:0/2273379939 <== mon.1 v1:172.21.15.103:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5bc0017f0 con 0x7fc5cc0a98f0 2024-03-25T19:25:05.296 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.296+0000 7fc5c9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:05.296 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.296+0000 7fc5c9d9b640 1 -- 172.21.15.83:0/2273379939 <== mon.1 v1:172.21.15.103:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fc5bc001c00 con 0x7fc5cc0a98f0 2024-03-25T19:25:05.296 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.296+0000 7fc5c9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:05.296 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.296+0000 7fc5c9d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:25:05.296 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.296+0000 7fc5c9d9b640 10 monclient: dump: 2024-03-25T19:25:05.296 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:25:05.296 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:25:05.296 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:05.296 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:05.296 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:05.296 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:25:05.296 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:25:05.297 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:25:05.297 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:25:05.297 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:05.297 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.296+0000 7fc5d100e640 10 monclient: get_monmap_and_config success 2024-03-25T19:25:05.298 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.296+0000 7fc5d100e640 4 set_mon_vals no callback set 2024-03-25T19:25:05.298 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.297+0000 7fc5d100e640 10 monclient: shutdown 2024-03-25T19:25:05.298 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.297+0000 7fc5d100e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:05.298 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.297+0000 7fc5d100e640 1 -- 172.21.15.83:0/2273379939 >> v1:172.21.15.103:6789/0 conn(0x7fc5cc0a98f0 legacy=0x7fc5cc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:05.298 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.297+0000 7fc5d100e640 1 -- 172.21.15.83:0/2273379939 shutdown_connections 2024-03-25T19:25:05.298 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.297+0000 7fc5d100e640 1 -- 172.21.15.83:0/2273379939 >> 172.21.15.83:0/2273379939 conn(0x7fc5cc0b7ea0 msgr2=0x7fc5cc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:05.298 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.297+0000 7fc5d100e640 1 -- 172.21.15.83:0/2273379939 shutdown_connections 2024-03-25T19:25:05.298 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.298+0000 7fc5d100e640 1 -- 172.21.15.83:0/2273379939 wait complete. 2024-03-25T19:25:05.300 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.298+0000 7fc5d100e640 10 monclient: build_initial_monmap 2024-03-25T19:25:05.300 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.298+0000 7fc5d100e640 10 monclient: monmap: 2024-03-25T19:25:05.300 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:25:05.300 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:25:05.300 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:25:05.300 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:25:05.300 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:05.300 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:25:05.300 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:25:05.300 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:25:05.301 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:25:05.301 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:05.301 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.299+0000 7fc5d100e640 1 librados: starting msgr at 2024-03-25T19:25:05.301 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.299+0000 7fc5d100e640 1 librados: starting objecter 2024-03-25T19:25:05.301 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.299+0000 7fc5d100e640 1 Processor -- start 2024-03-25T19:25:05.301 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.299+0000 7fc5d100e640 1 -- start start 2024-03-25T19:25:05.301 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.299+0000 7fc5d100e640 1 librados: setting wanted keys 2024-03-25T19:25:05.301 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.299+0000 7fc5d100e640 1 librados: calling monclient init 2024-03-25T19:25:05.301 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.299+0000 7fc5d100e640 10 monclient: init 2024-03-25T19:25:05.301 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.300+0000 7fc5d100e640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:05.301 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.300+0000 7fc5d100e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:25:05.301 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.300+0000 7fc5d100e640 10 monclient(hunting): picked mon.noname-c con 0x7fc5cc0a98f0 addr v1:172.21.15.83:6790/0 2024-03-25T19:25:05.301 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.300+0000 7fc5d100e640 10 monclient(hunting): picked mon.noname-a con 0x7fc5cc13f710 addr v1:172.21.15.83:6789/0 2024-03-25T19:25:05.302 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.300+0000 7fc5d100e640 10 monclient(hunting): picked mon.noname-b con 0x7fc5cc1430c0 addr v1:172.21.15.103:6789/0 2024-03-25T19:25:05.302 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.300+0000 7fc5d100e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5cc137ab0 con 0x7fc5cc13f710 2024-03-25T19:25:05.302 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.300+0000 7fc5d100e640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5cc138bd0 con 0x7fc5cc1430c0 2024-03-25T19:25:05.302 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.300+0000 7fc5d100e640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5cc138e40 con 0x7fc5cc0a98f0 2024-03-25T19:25:05.302 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.300+0000 7fc5d100e640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:05.302 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.300+0000 7fc5d100e640 10 monclient(hunting): authenticate will time out at 6362.572327s 2024-03-25T19:25:05.308 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.308+0000 7fc5ca59c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fc5cc13f710 0x7fc5cc1df270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:55954/0 (socket says 172.21.15.83:55954) 2024-03-25T19:25:05.308 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.308+0000 7fc5ca59c640 1 -- 172.21.15.83:0/1650575998 learned_addr learned my addr 172.21.15.83:0/1650575998 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:25:05.310 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.310+0000 7fc5b37fe640 10 client.?.objecter ms_handle_connect 0x7fc5cc13f710 2024-03-25T19:25:05.310 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.310+0000 7fc5b37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:05.311 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.311+0000 7fc5b37fe640 10 client.?.objecter ms_handle_connect 0x7fc5cc1430c0 2024-03-25T19:25:05.311 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.311+0000 7fc5b37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:05.321 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.321+0000 7fc5b37fe640 10 client.?.objecter ms_handle_connect 0x7fc5cc0a98f0 2024-03-25T19:25:05.321 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.321+0000 7fc5b37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:05.327 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.328+0000 7fc5b37fe640 1 -- 172.21.15.83:0/1650575998 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 473817505 0 0) 0x7fc5cc137ab0 con 0x7fc5cc13f710 2024-03-25T19:25:05.328 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.328+0000 7fc5b37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:25:05.328 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.328+0000 7fc5b37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:05.328 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.328+0000 7fc5b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:05.328 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.328+0000 7fc5b37fe640 10 monclient(hunting): my global_id is 4461 2024-03-25T19:25:05.328 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.328+0000 7fc5b37fe640 1 -- 172.21.15.83:0/1650575998 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5b8004eb0 con 0x7fc5cc13f710 2024-03-25T19:25:05.414 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.414+0000 7fc5b37fe640 1 -- 172.21.15.83:0/1650575998 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3851847020 0 0) 0x7fc5b8004eb0 con 0x7fc5cc13f710 2024-03-25T19:25:05.414 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.414+0000 7fc5b37fe640 1 -- 172.21.15.83:0/1650575998 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5cc137ab0 con 0x7fc5cc13f710 2024-03-25T19:25:05.415 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.414+0000 7fc5b37fe640 1 -- 172.21.15.83:0/1650575998 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fc5c0001a10 con 0x7fc5cc13f710 2024-03-25T19:25:05.415 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.414+0000 7fc5b37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:05.415 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.414+0000 7fc5b37fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:25:05.415 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.414+0000 7fc5b37fe640 10 monclient(hunting): dump: 2024-03-25T19:25:05.415 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:25:05.415 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:25:05.415 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:05.415 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:05.415 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:05.415 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:25:05.415 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:25:05.415 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:25:05.415 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:25:05.416 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:05.416 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.414+0000 7fc5b37fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:25:05.429 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.429+0000 7fc5b37fe640 1 -- 172.21.15.83:0/1650575998 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3079529213 0 0) 0x7fc5cc137ab0 con 0x7fc5cc13f710 2024-03-25T19:25:05.429 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.429+0000 7fc5b37fe640 1 -- 172.21.15.83:0/1650575998 >> v1:172.21.15.83:6790/0 conn(0x7fc5cc0a98f0 legacy=0x7fc5cc137f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:05.429 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.429+0000 7fc5b37fe640 1 -- 172.21.15.83:0/1650575998 >> v1:172.21.15.103:6789/0 conn(0x7fc5cc1430c0 legacy=0x7fc5cc1e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:05.430 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.429+0000 7fc5b37fe640 10 monclient: _finish_hunting 0 2024-03-25T19:25:05.430 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.429+0000 7fc5b37fe640 1 monclient: found mon.a 2024-03-25T19:25:05.430 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.429+0000 7fc5b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:05.430 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.429+0000 7fc5b37fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:25:05.430 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.429+0000 7fc5b37fe640 1 -- 172.21.15.83:0/1650575998 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5cc139120 con 0x7fc5cc13f710 2024-03-25T19:25:05.430 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.429+0000 7fc5b37fe640 10 monclient: _finish_auth 0 2024-03-25T19:25:05.430 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.429+0000 7fc5b37fe640 10 monclient: _check_auth_tickets 2024-03-25T19:25:05.430 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.429+0000 7fc5b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:05.430 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.429+0000 7fc5d100e640 5 monclient: authenticate success, global_id 4461 2024-03-25T19:25:05.430 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.429+0000 7fc5d100e640 10 monclient: _renew_subs 2024-03-25T19:25:05.430 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.429+0000 7fc5d100e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:25:05.431 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.429+0000 7fc5d100e640 1 -- 172.21.15.83:0/1650575998 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5cc139400 con 0x7fc5cc13f710 2024-03-25T19:25:05.431 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.429+0000 7fc5d100e640 10 client.4461.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:25:05.431 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.429+0000 7fc5d100e640 10 monclient: _renew_subs 2024-03-25T19:25:05.431 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.429+0000 7fc5d100e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:25:05.431 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.429+0000 7fc5d100e640 1 -- 172.21.15.83:0/1650575998 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5cc0b7880 con 0x7fc5cc13f710 2024-03-25T19:25:05.431 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.429+0000 7fc5d100e640 1 librados: init done 2024-03-25T19:25:05.431 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.430+0000 7fc5d100e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:05.432 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.430+0000 7fc5d100e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:05.432 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.430+0000 7fc5d100e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:25:05.432 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.430+0000 7fc5d100e640 1 -- 172.21.15.83:0/1650575998 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5cc137810 con 0x7fc5cc13f710 2024-03-25T19:25:05.684 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.684+0000 7fc5b37fe640 1 -- 172.21.15.83:0/1650575998 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5c0001cb0 con 0x7fc5cc13f710 2024-03-25T19:25:05.684 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.684+0000 7fc5b37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:05.684 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.684+0000 7fc5b37fe640 1 -- 172.21.15.83:0/1650575998 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fc5c0003a60 con 0x7fc5cc13f710 2024-03-25T19:25:05.684 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.684+0000 7fc5b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:05.684 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.684+0000 7fc5b37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:25:05.684 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.684+0000 7fc5b37fe640 10 monclient: dump: 2024-03-25T19:25:05.684 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:25:05.684 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:25:05.684 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:05.685 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:05.685 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:05.685 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:25:05.685 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:25:05.685 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:25:05.685 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:25:05.685 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:05.685 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.684+0000 7fc5c959a640 4 set_mon_vals no callback set 2024-03-25T19:25:05.686 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.685+0000 7fc5b37fe640 1 -- 172.21.15.83:0/1650575998 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7fc5c0003c80 con 0x7fc5cc13f710 2024-03-25T19:25:05.686 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.685+0000 7fc5cad9d640 10 monclient: get_auth_request con 0x7fc5b806dc70 auth_method 0 2024-03-25T19:25:05.686 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.686+0000 7fc5b37fe640 10 client.4461.objecter ms_handle_connect 0x7fc5b806dc70 2024-03-25T19:25:05.695 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.695+0000 7fc5b37fe640 1 -- 172.21.15.83:0/1650575998 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7fc5c00548a0 con 0x7fc5cc13f710 2024-03-25T19:25:05.697 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.695+0000 7fc5b37fe640 10 client.4461.objecter ms_dispatch 0x7fc5cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:25:05.697 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.695+0000 7fc5b37fe640 3 client.4461.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:25:05.697 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.695+0000 7fc5b37fe640 3 client.4461.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:25:05.697 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.695+0000 7fc5b37fe640 20 client.4461.objecter dump_active .. 0 homeless 2024-03-25T19:25:05.697 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.695+0000 7fc5b37fe640 1 -- 172.21.15.83:0/1650575998 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc5c00503b0 con 0x7fc5cc13f710 2024-03-25T19:25:05.697 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.695+0000 7fc5b37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:05.697 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.695+0000 7fc5b37fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:25:05.934 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.934+0000 7fc5d100e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:05.934 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.934+0000 7fc5d100e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:05.935 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.934+0000 7fc5d100e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:25:05.935 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.934+0000 7fc5d100e640 1 -- 172.21.15.83:0/1650575998 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc5cc137c60 con 0x7fc5cc13f710 2024-03-25T19:25:05.980 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.980+0000 7fc5b37fe640 1 -- 172.21.15.83:0/1650575998 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7fc5cc137c60 con 0x7fc5cc13f710 2024-03-25T19:25:05.980 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.980+0000 7fc5b37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:05.980 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.980+0000 7fc5b37fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:25:05.980 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:25:05.981 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:25:05.985 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.983+0000 7fc5d100e640 10 librados: watch_flush enter 2024-03-25T19:25:05.985 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.983+0000 7fc5d100e640 10 librados: watch_flush exit 2024-03-25T19:25:05.985 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.983+0000 7fc5d100e640 20 client.4461.objecter shutdown clearing up homeless session... 2024-03-25T19:25:05.985 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.983+0000 7fc5d100e640 10 client.4461.objecter successfully canceled tick 2024-03-25T19:25:05.986 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.983+0000 7fc5d100e640 1 -- 172.21.15.83:0/1650575998 >> v1:172.21.15.103:6800/345072327 conn(0x7fc5b806dc70 legacy=0x7fc5b8070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:05.986 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.983+0000 7fc5d100e640 10 monclient: shutdown 2024-03-25T19:25:05.986 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.983+0000 7fc5d100e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:05.986 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.983+0000 7fc5d100e640 1 -- 172.21.15.83:0/1650575998 >> v1:172.21.15.83:6789/0 conn(0x7fc5cc13f710 legacy=0x7fc5cc1df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:05.986 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.983+0000 7fc5d100e640 1 -- 172.21.15.83:0/1650575998 shutdown_connections 2024-03-25T19:25:05.986 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.983+0000 7fc5d100e640 1 -- 172.21.15.83:0/1650575998 >> 172.21.15.83:0/1650575998 conn(0x7fc5cc0b7ea0 msgr2=0x7fc5cc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:05.986 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.983+0000 7fc5d100e640 1 -- 172.21.15.83:0/1650575998 shutdown_connections 2024-03-25T19:25:05.986 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.983+0000 7fc5d100e640 1 -- 172.21.15.83:0/1650575998 wait complete. 2024-03-25T19:25:05.986 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:05.983+0000 7fc5d100e640 1 librados: shutdown 2024-03-25T19:25:05.995 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:25:12.801 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:25:12.919 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.920+0000 7f92aa777640 10 monclient: get_monmap_and_config 2024-03-25T19:25:12.919 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.920+0000 7f92aa777640 10 monclient: build_initial_monmap 2024-03-25T19:25:12.920 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.920+0000 7f92aa777640 10 monclient: monmap: 2024-03-25T19:25:12.920 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:25:12.920 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:25:12.920 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:25:12.920978+0000 2024-03-25T19:25:12.920 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:25:12.920978+0000 2024-03-25T19:25:12.920 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:12.920 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:25:12.920 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:25:12.920 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:25:12.920 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:25:12.920 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:12.921 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.921+0000 7f92aa777640 1 Processor -- start 2024-03-25T19:25:12.921 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.921+0000 7f92aa777640 1 -- start start 2024-03-25T19:25:12.922 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.921+0000 7f92aa777640 10 monclient: init 2024-03-25T19:25:12.922 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.921+0000 7f92aa777640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:12.922 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.921+0000 7f92aa777640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:25:12.922 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.922+0000 7f92aa777640 10 monclient(hunting): picked mon.noname-a con 0x7f92a40a98f0 addr v1:172.21.15.83:6789/0 2024-03-25T19:25:12.922 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.922+0000 7f92aa777640 10 monclient(hunting): picked mon.noname-b con 0x7f92a413f710 addr v1:172.21.15.83:6790/0 2024-03-25T19:25:12.922 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.922+0000 7f92aa777640 10 monclient(hunting): picked mon.noname-c con 0x7f92a41430c0 addr v1:172.21.15.103:6789/0 2024-03-25T19:25:12.922 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.922+0000 7f92aa777640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92a4142520 con 0x7f92a40a98f0 2024-03-25T19:25:12.922 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.922+0000 7f92aa777640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92a40b28a0 con 0x7f92a41430c0 2024-03-25T19:25:12.922 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.922+0000 7f92aa777640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92a40b2a30 con 0x7f92a413f710 2024-03-25T19:25:12.922 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.922+0000 7f92aa777640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:12.922 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.922+0000 7f92aa777640 10 monclient(hunting): authenticate will time out at 6370.193814s 2024-03-25T19:25:12.930 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.930+0000 7f92a3fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f92a40a98f0 0x7f92a40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:50688/0 (socket says 172.21.15.83:50688) 2024-03-25T19:25:12.930 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.930+0000 7f92a3fff640 1 -- 172.21.15.83:0/22543015 learned_addr learned my addr 172.21.15.83:0/22543015 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:25:12.942 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.943+0000 7f92a2ffd640 1 -- 172.21.15.83:0/22543015 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1537278684 0 0) 0x7f92a4142520 con 0x7f92a40a98f0 2024-03-25T19:25:12.942 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.943+0000 7f92a2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:25:12.942 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.943+0000 7f92a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:12.942 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.943+0000 7f92a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:12.943 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.943+0000 7f92a2ffd640 10 monclient(hunting): my global_id is 4464 2024-03-25T19:25:12.943 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.943+0000 7f92a2ffd640 1 -- 172.21.15.83:0/22543015 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f928c003520 con 0x7f92a40a98f0 2024-03-25T19:25:12.950 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.950+0000 7f92a2ffd640 1 -- 172.21.15.83:0/22543015 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3515843952 0 0) 0x7f928c003520 con 0x7f92a40a98f0 2024-03-25T19:25:12.951 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.950+0000 7f92a2ffd640 1 -- 172.21.15.83:0/22543015 >> v1:172.21.15.83:6790/0 conn(0x7f92a413f710 legacy=0x7f92a41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:12.951 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.950+0000 7f92a2ffd640 1 -- 172.21.15.83:0/22543015 >> v1:172.21.15.103:6789/0 conn(0x7f92a41430c0 legacy=0x7f92a414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:12.951 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.950+0000 7f92a2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:25:12.951 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.950+0000 7f92a2ffd640 1 monclient: found mon.noname-a 2024-03-25T19:25:12.951 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.950+0000 7f92a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:12.951 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.950+0000 7f92a2ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.83:6789/0 2024-03-25T19:25:12.951 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.950+0000 7f92a2ffd640 1 -- 172.21.15.83:0/22543015 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92a40b44c0 con 0x7f92a40a98f0 2024-03-25T19:25:12.951 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.950+0000 7f92a2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:25:12.951 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.950+0000 7f92a2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:25:12.951 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.950+0000 7f92a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:12.951 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.950+0000 7f92a2ffd640 1 -- 172.21.15.83:0/22543015 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f9294001690 con 0x7f92a40a98f0 2024-03-25T19:25:12.951 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.950+0000 7f92a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:12.951 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.950+0000 7f92a2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:25:12.951 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.950+0000 7f92a2ffd640 10 monclient: dump: 2024-03-25T19:25:12.952 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:25:12.952 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:25:12.952 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:12.952 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:12.952 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:12.952 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:25:12.952 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:25:12.952 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:25:12.952 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:25:12.952 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:12.952 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.950+0000 7f92aa777640 5 monclient: authenticate success, global_id 4464 2024-03-25T19:25:12.952 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.950+0000 7f92aa777640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:25:12.973 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.974+0000 7f92a2ffd640 1 -- 172.21.15.83:0/22543015 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92940017f0 con 0x7f92a40a98f0 2024-03-25T19:25:12.973 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.974+0000 7f92a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:12.973 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.974+0000 7f92a2ffd640 1 -- 172.21.15.83:0/22543015 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f9294001c00 con 0x7f92a40a98f0 2024-03-25T19:25:12.974 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.974+0000 7f92aa777640 10 monclient: get_monmap_and_config success 2024-03-25T19:25:12.974 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.974+0000 7f92aa777640 4 set_mon_vals no callback set 2024-03-25T19:25:12.974 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.974+0000 7f92a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:12.974 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.974+0000 7f92a2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:25:12.974 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.974+0000 7f92a2ffd640 10 monclient: dump: 2024-03-25T19:25:12.974 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:25:12.974 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:25:12.974 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:12.974 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:12.974 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:12.974 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:25:12.974 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:25:12.974 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:25:12.974 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:25:12.975 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:12.976 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.974+0000 7f92aa777640 10 monclient: shutdown 2024-03-25T19:25:12.976 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.974+0000 7f92aa777640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:12.976 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.974+0000 7f92aa777640 1 -- 172.21.15.83:0/22543015 >> v1:172.21.15.83:6789/0 conn(0x7f92a40a98f0 legacy=0x7f92a40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:12.976 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.975+0000 7f92aa777640 1 -- 172.21.15.83:0/22543015 shutdown_connections 2024-03-25T19:25:12.976 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.975+0000 7f92aa777640 1 -- 172.21.15.83:0/22543015 >> 172.21.15.83:0/22543015 conn(0x7f92a40b7ea0 msgr2=0x7f92a40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:12.976 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.975+0000 7f92aa777640 1 -- 172.21.15.83:0/22543015 shutdown_connections 2024-03-25T19:25:12.976 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.975+0000 7f92aa777640 1 -- 172.21.15.83:0/22543015 wait complete. 2024-03-25T19:25:12.976 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.975+0000 7f92aa777640 10 monclient: build_initial_monmap 2024-03-25T19:25:12.976 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.975+0000 7f92aa777640 10 monclient: monmap: 2024-03-25T19:25:12.976 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:25:12.976 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:25:12.976 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:25:12.977 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:25:12.977 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:12.977 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:25:12.977 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:25:12.977 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:25:12.977 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:25:12.977 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:12.977 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.976+0000 7f92aa777640 1 librados: starting msgr at 2024-03-25T19:25:12.977 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.976+0000 7f92aa777640 1 librados: starting objecter 2024-03-25T19:25:12.977 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.976+0000 7f92aa777640 1 Processor -- start 2024-03-25T19:25:12.977 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.976+0000 7f92aa777640 1 -- start start 2024-03-25T19:25:12.977 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.976+0000 7f92aa777640 1 librados: setting wanted keys 2024-03-25T19:25:12.977 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.976+0000 7f92aa777640 1 librados: calling monclient init 2024-03-25T19:25:12.977 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.976+0000 7f92aa777640 10 monclient: init 2024-03-25T19:25:12.978 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.976+0000 7f92aa777640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:12.979 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.976+0000 7f92aa777640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:25:12.979 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.977+0000 7f92aa777640 10 monclient(hunting): picked mon.noname-c con 0x7f92a40a98f0 addr v1:172.21.15.83:6790/0 2024-03-25T19:25:12.979 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.977+0000 7f92aa777640 10 monclient(hunting): picked mon.noname-a con 0x7f92a413f710 addr v1:172.21.15.83:6789/0 2024-03-25T19:25:12.979 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.977+0000 7f92aa777640 10 monclient(hunting): picked mon.noname-b con 0x7f92a41430c0 addr v1:172.21.15.103:6789/0 2024-03-25T19:25:12.979 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.977+0000 7f92aa777640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92a40abd30 con 0x7f92a413f710 2024-03-25T19:25:12.979 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.977+0000 7f92aa777640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92a41d77a0 con 0x7f92a41430c0 2024-03-25T19:25:12.979 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.977+0000 7f92aa777640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92a41d51f0 con 0x7f92a40a98f0 2024-03-25T19:25:12.979 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.977+0000 7f92aa777640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:12.979 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.977+0000 7f92aa777640 10 monclient(hunting): authenticate will time out at 6370.248835s 2024-03-25T19:25:12.980 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.981+0000 7f92a8d14640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f92a41430c0 0x7f92a41e1ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:46662/0 (socket says 172.21.15.83:46662) 2024-03-25T19:25:12.980 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.981+0000 7f92a8d14640 1 -- 172.21.15.83:0/2913527893 learned_addr learned my addr 172.21.15.83:0/2913527893 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:25:12.983 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.983+0000 7f92a0ff9640 10 client.?.objecter ms_handle_connect 0x7f92a40a98f0 2024-03-25T19:25:12.983 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.983+0000 7f92a0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:12.985 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.985+0000 7f92a0ff9640 10 client.?.objecter ms_handle_connect 0x7f92a41430c0 2024-03-25T19:25:12.985 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.985+0000 7f92a0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:12.993 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.994+0000 7f92a0ff9640 1 -- 172.21.15.83:0/2913527893 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4226848469 0 0) 0x7f92a41d77a0 con 0x7f92a41430c0 2024-03-25T19:25:12.993 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.994+0000 7f92a0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:25:12.994 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.994+0000 7f92a0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:12.994 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.994+0000 7f92a0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:12.994 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.994+0000 7f92a0ff9640 10 monclient(hunting): my global_id is 4477 2024-03-25T19:25:12.994 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.994+0000 7f92a0ff9640 1 -- 172.21.15.83:0/2913527893 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9290004eb0 con 0x7f92a41430c0 2024-03-25T19:25:12.994 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.994+0000 7f92a0ff9640 10 client.?.objecter ms_handle_connect 0x7f92a413f710 2024-03-25T19:25:12.994 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.994+0000 7f92a0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:12.997 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.997+0000 7f92a0ff9640 1 -- 172.21.15.83:0/2913527893 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3580539984 0 0) 0x7f92a41d51f0 con 0x7f92a40a98f0 2024-03-25T19:25:12.997 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.997+0000 7f92a0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:25:12.997 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.997+0000 7f92a0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:12.997 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.997+0000 7f92a0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:12.997 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.997+0000 7f92a0ff9640 10 monclient(hunting): my global_id is 4484 2024-03-25T19:25:12.997 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:12.997+0000 7f92a0ff9640 1 -- 172.21.15.83:0/2913527893 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92a41d77a0 con 0x7f92a40a98f0 2024-03-25T19:25:13.006 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.006+0000 7f92a0ff9640 1 -- 172.21.15.83:0/2913527893 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 919327616 0 0) 0x7f92a40abd30 con 0x7f92a413f710 2024-03-25T19:25:13.006 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.006+0000 7f92a0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:25:13.006 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.006+0000 7f92a0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:13.006 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.006+0000 7f92a0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:13.006 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.006+0000 7f92a0ff9640 10 monclient(hunting): my global_id is 4467 2024-03-25T19:25:13.007 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.006+0000 7f92a0ff9640 1 -- 172.21.15.83:0/2913527893 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92a41d51f0 con 0x7f92a413f710 2024-03-25T19:25:13.007 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.006+0000 7f92a0ff9640 1 -- 172.21.15.83:0/2913527893 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2119367900 0 0) 0x7f92a41d77a0 con 0x7f92a40a98f0 2024-03-25T19:25:13.007 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.006+0000 7f92a0ff9640 1 -- 172.21.15.83:0/2913527893 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92a40abd30 con 0x7f92a40a98f0 2024-03-25T19:25:13.007 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.006+0000 7f92a0ff9640 1 -- 172.21.15.83:0/2913527893 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f9294001b40 con 0x7f92a40a98f0 2024-03-25T19:25:13.007 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.006+0000 7f92a0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:13.007 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.006+0000 7f92a0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:25:13.007 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.006+0000 7f92a0ff9640 10 monclient(hunting): dump: 2024-03-25T19:25:13.007 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:25:13.007 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:25:13.007 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:13.007 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:13.007 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:13.007 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:25:13.007 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:25:13.007 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:25:13.007 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:25:13.008 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:13.008 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.006+0000 7f92a0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:25:13.009 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.009+0000 7f92a0ff9640 1 -- 172.21.15.83:0/2913527893 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1957952611 0 0) 0x7f9290004eb0 con 0x7f92a41430c0 2024-03-25T19:25:13.009 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.009+0000 7f92a0ff9640 1 -- 172.21.15.83:0/2913527893 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92a41d77a0 con 0x7f92a41430c0 2024-03-25T19:25:13.009 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.009+0000 7f92a0ff9640 1 -- 172.21.15.83:0/2913527893 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f929c001c30 con 0x7f92a41430c0 2024-03-25T19:25:13.009 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.009+0000 7f92a0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:13.009 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.009+0000 7f92a0ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:25:13.009 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.009+0000 7f92a0ff9640 10 monclient(hunting): dump: 2024-03-25T19:25:13.009 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:25:13.009 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:25:13.009 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:13.010 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:13.010 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:13.010 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:25:13.010 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:25:13.010 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:25:13.010 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:25:13.010 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:13.021 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.021+0000 7f92a0ff9640 1 -- 172.21.15.83:0/2913527893 <== mon.1 v1:172.21.15.103:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 969168453 0 0) 0x7f92a41d77a0 con 0x7f92a41430c0 2024-03-25T19:25:13.021 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.021+0000 7f92a0ff9640 1 -- 172.21.15.83:0/2913527893 >> v1:172.21.15.83:6790/0 conn(0x7f92a40a98f0 legacy=0x7f92a41d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:13.021 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.021+0000 7f92a0ff9640 1 -- 172.21.15.83:0/2913527893 >> v1:172.21.15.83:6789/0 conn(0x7f92a413f710 legacy=0x7f92a4138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:13.021 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.021+0000 7f92a0ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:25:13.021 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.021+0000 7f92a0ff9640 1 monclient: found mon.b 2024-03-25T19:25:13.021 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.021+0000 7f92a0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:13.021 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.021+0000 7f92a0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:25:13.021 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.021+0000 7f92a0ff9640 1 -- 172.21.15.83:0/2913527893 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92a41d83d0 con 0x7f92a41430c0 2024-03-25T19:25:13.021 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.021+0000 7f92a0ff9640 10 monclient: _finish_auth 0 2024-03-25T19:25:13.021 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.021+0000 7f92a0ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:25:13.022 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.021+0000 7f92a0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:13.022 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.021+0000 7f92aa777640 5 monclient: authenticate success, global_id 4477 2024-03-25T19:25:13.022 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.021+0000 7f92aa777640 10 monclient: _renew_subs 2024-03-25T19:25:13.022 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.021+0000 7f92aa777640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:25:13.022 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.021+0000 7f92aa777640 1 -- 172.21.15.83:0/2913527893 --> v1:172.21.15.103:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92a4138a50 con 0x7f92a41430c0 2024-03-25T19:25:13.022 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.021+0000 7f92aa777640 10 client.4477.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:25:13.022 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.021+0000 7f92aa777640 10 monclient: _renew_subs 2024-03-25T19:25:13.022 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.021+0000 7f92aa777640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:25:13.022 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.021+0000 7f92aa777640 1 -- 172.21.15.83:0/2913527893 --> v1:172.21.15.103:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f92a41e35c0 con 0x7f92a41430c0 2024-03-25T19:25:13.022 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.021+0000 7f92aa777640 1 librados: init done 2024-03-25T19:25:13.023 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.022+0000 7f92aa777640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:13.023 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.022+0000 7f92aa777640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:13.023 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.022+0000 7f92aa777640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:25:13.023 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.022+0000 7f92aa777640 1 -- 172.21.15.83:0/2913527893 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92a40627b0 con 0x7f92a41430c0 2024-03-25T19:25:13.311 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.311+0000 7f92a0ff9640 1 -- 172.21.15.83:0/2913527893 <== mon.1 v1:172.21.15.103:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f929c003990 con 0x7f92a41430c0 2024-03-25T19:25:13.311 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.311+0000 7f92a0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:13.311 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.311+0000 7f92a0ff9640 1 -- 172.21.15.83:0/2913527893 <== mon.1 v1:172.21.15.103:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f929c002100 con 0x7f92a41430c0 2024-03-25T19:25:13.311 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.311+0000 7f92a0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:13.311 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.311+0000 7f92a0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:25:13.311 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.311+0000 7f92a0ff9640 10 monclient: dump: 2024-03-25T19:25:13.311 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:25:13.312 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:25:13.312 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:13.312 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:13.312 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:13.312 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:25:13.312 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:25:13.312 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:25:13.312 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:25:13.312 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:13.312 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.311+0000 7f92a27fc640 4 set_mon_vals no callback set 2024-03-25T19:25:13.313 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.312+0000 7f92a0ff9640 1 -- 172.21.15.83:0/2913527893 <== mon.1 v1:172.21.15.103:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7f929c003e60 con 0x7f92a41430c0 2024-03-25T19:25:13.313 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.312+0000 7f92a3fff640 10 monclient: get_auth_request con 0x7f929006dfd0 auth_method 0 2024-03-25T19:25:13.313 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.313+0000 7f92a0ff9640 10 client.4477.objecter ms_handle_connect 0x7f929006dfd0 2024-03-25T19:25:13.317 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.317+0000 7f92a0ff9640 1 -- 172.21.15.83:0/2913527893 <== mon.1 v1:172.21.15.103:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7f929c054a30 con 0x7f92a41430c0 2024-03-25T19:25:13.317 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.317+0000 7f92a0ff9640 10 client.4477.objecter ms_dispatch 0x7f92a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:25:13.317 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.317+0000 7f92a0ff9640 3 client.4477.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:25:13.317 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.317+0000 7f92a0ff9640 3 client.4477.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:25:13.317 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.317+0000 7f92a0ff9640 20 client.4477.objecter dump_active .. 0 homeless 2024-03-25T19:25:13.318 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.318+0000 7f92a0ff9640 1 -- 172.21.15.83:0/2913527893 <== mon.1 v1:172.21.15.103:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f929c050540 con 0x7f92a41430c0 2024-03-25T19:25:13.318 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.318+0000 7f92a0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:13.318 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.318+0000 7f92a0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:25:13.545 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.545+0000 7f92aa777640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:13.545 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.545+0000 7f92aa777640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:13.545 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.545+0000 7f92aa777640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:25:13.545 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.545+0000 7f92aa777640 1 -- 172.21.15.83:0/2913527893 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f92a4074cc0 con 0x7f92a41430c0 2024-03-25T19:25:13.563 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.563+0000 7f92a0ff9640 1 -- 172.21.15.83:0/2913527893 <== mon.1 v1:172.21.15.103:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7f929c052470 con 0x7f92a41430c0 2024-03-25T19:25:13.563 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.563+0000 7f92a0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:13.563 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.563+0000 7f92a0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:25:13.563 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:25:13.563 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:25:13.565 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.565+0000 7f92aa777640 10 librados: watch_flush enter 2024-03-25T19:25:13.566 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.565+0000 7f92aa777640 10 librados: watch_flush exit 2024-03-25T19:25:13.566 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.565+0000 7f92aa777640 20 client.4477.objecter shutdown clearing up homeless session... 2024-03-25T19:25:13.566 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.565+0000 7f92aa777640 10 client.4477.objecter successfully canceled tick 2024-03-25T19:25:13.566 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.565+0000 7f92aa777640 1 -- 172.21.15.83:0/2913527893 >> v1:172.21.15.103:6800/345072327 conn(0x7f929006dfd0 legacy=0x7f9290070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:13.566 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.565+0000 7f92aa777640 10 monclient: shutdown 2024-03-25T19:25:13.566 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.565+0000 7f92aa777640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:13.566 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.565+0000 7f92aa777640 1 -- 172.21.15.83:0/2913527893 >> v1:172.21.15.103:6789/0 conn(0x7f92a41430c0 legacy=0x7f92a41e1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:13.566 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.565+0000 7f92aa777640 1 -- 172.21.15.83:0/2913527893 shutdown_connections 2024-03-25T19:25:13.566 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.565+0000 7f92aa777640 1 -- 172.21.15.83:0/2913527893 >> 172.21.15.83:0/2913527893 conn(0x7f92a40b7ea0 msgr2=0x7f92a4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:13.566 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.565+0000 7f92aa777640 1 -- 172.21.15.83:0/2913527893 shutdown_connections 2024-03-25T19:25:13.566 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.565+0000 7f92aa777640 1 -- 172.21.15.83:0/2913527893 wait complete. 2024-03-25T19:25:13.566 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:13.565+0000 7f92aa777640 1 librados: shutdown 2024-03-25T19:25:13.576 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:25:20.383 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:25:20.503 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.503+0000 7f7fffeed640 10 monclient: get_monmap_and_config 2024-03-25T19:25:20.503 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.503+0000 7f7fffeed640 10 monclient: build_initial_monmap 2024-03-25T19:25:20.503 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.503+0000 7f7fffeed640 10 monclient: monmap: 2024-03-25T19:25:20.503 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:25:20.503 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:25:20.503 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:25:20.504329+0000 2024-03-25T19:25:20.503 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:25:20.504329+0000 2024-03-25T19:25:20.504 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:20.504 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:25:20.504 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:25:20.504 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:25:20.504 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:25:20.504 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:20.505 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.504+0000 7f7fffeed640 1 Processor -- start 2024-03-25T19:25:20.505 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.504+0000 7f7fffeed640 1 -- start start 2024-03-25T19:25:20.505 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.504+0000 7f7fffeed640 10 monclient: init 2024-03-25T19:25:20.505 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.505+0000 7f7fffeed640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:20.505 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.505+0000 7f7fffeed640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:25:20.505 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.505+0000 7f7fffeed640 10 monclient(hunting): picked mon.noname-c con 0x7f7ff80a98f0 addr v1:172.21.15.103:6789/0 2024-03-25T19:25:20.505 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.505+0000 7f7fffeed640 10 monclient(hunting): picked mon.noname-b con 0x7f7ff813f710 addr v1:172.21.15.83:6790/0 2024-03-25T19:25:20.506 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.505+0000 7f7fffeed640 10 monclient(hunting): picked mon.noname-a con 0x7f7ff81430c0 addr v1:172.21.15.83:6789/0 2024-03-25T19:25:20.506 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.505+0000 7f7fffeed640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff8142520 con 0x7f7ff81430c0 2024-03-25T19:25:20.506 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.505+0000 7f7fffeed640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff80b28a0 con 0x7f7ff80a98f0 2024-03-25T19:25:20.506 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.505+0000 7f7fffeed640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff80b2a30 con 0x7f7ff813f710 2024-03-25T19:25:20.506 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.505+0000 7f7fffeed640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:20.506 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.505+0000 7f7fffeed640 10 monclient(hunting): authenticate will time out at 6377.777106s 2024-03-25T19:25:20.513 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.513+0000 7f7ffdc89640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f7ff80a98f0 0x7f7ff80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:60820/0 (socket says 172.21.15.83:60820) 2024-03-25T19:25:20.513 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.513+0000 7f7ffdc89640 1 -- 172.21.15.83:0/4141601943 learned_addr learned my addr 172.21.15.83:0/4141601943 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:25:20.525 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.525+0000 7f7ffcc87640 1 -- 172.21.15.83:0/4141601943 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 556145267 0 0) 0x7f7ff80b28a0 con 0x7f7ff80a98f0 2024-03-25T19:25:20.525 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.525+0000 7f7ffcc87640 10 monclient(hunting): _negotiate 2024-03-25T19:25:20.525 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.525+0000 7f7ffcc87640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:20.525 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.525+0000 7f7ffcc87640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:20.525 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.525+0000 7f7ffcc87640 10 monclient(hunting): my global_id is 4480 2024-03-25T19:25:20.525 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.525+0000 7f7ffcc87640 1 -- 172.21.15.83:0/4141601943 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fe4003520 con 0x7f7ff80a98f0 2024-03-25T19:25:20.537 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.537+0000 7f7ffcc87640 1 -- 172.21.15.83:0/4141601943 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 29618044 0 0) 0x7f7ff8142520 con 0x7f7ff81430c0 2024-03-25T19:25:20.537 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.537+0000 7f7ffcc87640 10 monclient(hunting): _negotiate 2024-03-25T19:25:20.537 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.537+0000 7f7ffcc87640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:20.537 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.537+0000 7f7ffcc87640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:20.537 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.537+0000 7f7ffcc87640 10 monclient(hunting): my global_id is 4470 2024-03-25T19:25:20.537 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.537+0000 7f7ffcc87640 1 -- 172.21.15.83:0/4141601943 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ff80b28a0 con 0x7f7ff81430c0 2024-03-25T19:25:20.545 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.545+0000 7f7ffcc87640 1 -- 172.21.15.83:0/4141601943 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1588589650 0 0) 0x7f7fe4003520 con 0x7f7ff80a98f0 2024-03-25T19:25:20.545 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.545+0000 7f7ffcc87640 1 -- 172.21.15.83:0/4141601943 >> v1:172.21.15.83:6790/0 conn(0x7f7ff813f710 legacy=0x7f7ff81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:20.545 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.545+0000 7f7ffcc87640 1 -- 172.21.15.83:0/4141601943 >> v1:172.21.15.83:6789/0 conn(0x7f7ff81430c0 legacy=0x7f7ff814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:20.545 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.545+0000 7f7ffcc87640 10 monclient: _finish_hunting 0 2024-03-25T19:25:20.545 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.545+0000 7f7ffcc87640 1 monclient: found mon.noname-c 2024-03-25T19:25:20.545 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.545+0000 7f7ffcc87640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:20.546 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.545+0000 7f7ffcc87640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.103:6789/0 2024-03-25T19:25:20.546 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.545+0000 7f7ffcc87640 1 -- 172.21.15.83:0/4141601943 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff80b44c0 con 0x7f7ff80a98f0 2024-03-25T19:25:20.546 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.545+0000 7f7ffcc87640 10 monclient: _finish_auth 0 2024-03-25T19:25:20.546 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.545+0000 7f7ffcc87640 10 monclient: _check_auth_tickets 2024-03-25T19:25:20.546 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.545+0000 7f7ffcc87640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:20.546 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.545+0000 7f7ffcc87640 1 -- 172.21.15.83:0/4141601943 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f7ff4001690 con 0x7f7ff80a98f0 2024-03-25T19:25:20.546 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.545+0000 7f7ffcc87640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:20.546 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.545+0000 7f7ffcc87640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:25:20.546 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.545+0000 7f7ffcc87640 10 monclient: dump: 2024-03-25T19:25:20.546 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:25:20.547 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:25:20.547 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:20.547 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:20.547 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:20.547 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:25:20.547 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:25:20.547 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:25:20.547 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:25:20.547 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:20.547 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.545+0000 7f7fffeed640 5 monclient: authenticate success, global_id 4480 2024-03-25T19:25:20.547 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.545+0000 7f7fffeed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:25:20.558 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.558+0000 7f7ffcc87640 1 -- 172.21.15.83:0/4141601943 <== mon.1 v1:172.21.15.103:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ff40017f0 con 0x7f7ff80a98f0 2024-03-25T19:25:20.558 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.558+0000 7f7ffcc87640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:20.558 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.558+0000 7f7ffcc87640 1 -- 172.21.15.83:0/4141601943 <== mon.1 v1:172.21.15.103:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f7ff4001c00 con 0x7f7ff80a98f0 2024-03-25T19:25:20.558 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.558+0000 7f7fffeed640 10 monclient: get_monmap_and_config success 2024-03-25T19:25:20.558 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.558+0000 7f7fffeed640 4 set_mon_vals no callback set 2024-03-25T19:25:20.558 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.558+0000 7f7ffcc87640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:20.559 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.558+0000 7f7ffcc87640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:25:20.559 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.558+0000 7f7ffcc87640 10 monclient: dump: 2024-03-25T19:25:20.559 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:25:20.559 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:25:20.559 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:20.559 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:20.559 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:20.559 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:25:20.559 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:25:20.560 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:25:20.560 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:25:20.560 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:20.562 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.559+0000 7f7fffeed640 10 monclient: shutdown 2024-03-25T19:25:20.562 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.559+0000 7f7fffeed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:20.563 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.559+0000 7f7fffeed640 1 -- 172.21.15.83:0/4141601943 >> v1:172.21.15.103:6789/0 conn(0x7f7ff80a98f0 legacy=0x7f7ff80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:20.563 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.559+0000 7f7fffeed640 1 -- 172.21.15.83:0/4141601943 shutdown_connections 2024-03-25T19:25:20.563 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.559+0000 7f7fffeed640 1 -- 172.21.15.83:0/4141601943 >> 172.21.15.83:0/4141601943 conn(0x7f7ff80b7ea0 msgr2=0x7f7ff80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:20.563 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.560+0000 7f7fffeed640 1 -- 172.21.15.83:0/4141601943 shutdown_connections 2024-03-25T19:25:20.563 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.560+0000 7f7fffeed640 1 -- 172.21.15.83:0/4141601943 wait complete. 2024-03-25T19:25:20.563 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.560+0000 7f7fffeed640 10 monclient: build_initial_monmap 2024-03-25T19:25:20.563 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.560+0000 7f7fffeed640 10 monclient: monmap: 2024-03-25T19:25:20.563 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:25:20.563 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:25:20.563 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:25:20.563 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:25:20.563 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:20.563 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:25:20.563 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:25:20.563 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:25:20.563 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:25:20.564 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:20.564 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.561+0000 7f7fffeed640 1 librados: starting msgr at 2024-03-25T19:25:20.564 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.561+0000 7f7fffeed640 1 librados: starting objecter 2024-03-25T19:25:20.564 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.561+0000 7f7fffeed640 1 Processor -- start 2024-03-25T19:25:20.564 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.561+0000 7f7fffeed640 1 -- start start 2024-03-25T19:25:20.564 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.561+0000 7f7fffeed640 1 librados: setting wanted keys 2024-03-25T19:25:20.564 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.561+0000 7f7fffeed640 1 librados: calling monclient init 2024-03-25T19:25:20.564 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.561+0000 7f7fffeed640 10 monclient: init 2024-03-25T19:25:20.564 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.562+0000 7f7fffeed640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:20.564 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.562+0000 7f7fffeed640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:25:20.564 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.562+0000 7f7fffeed640 10 monclient(hunting): picked mon.noname-b con 0x7f7ff80a98f0 addr v1:172.21.15.103:6789/0 2024-03-25T19:25:20.564 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.562+0000 7f7fffeed640 10 monclient(hunting): picked mon.noname-c con 0x7f7ff813f710 addr v1:172.21.15.83:6790/0 2024-03-25T19:25:20.564 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.562+0000 7f7fffeed640 10 monclient(hunting): picked mon.noname-a con 0x7f7ff81430c0 addr v1:172.21.15.83:6789/0 2024-03-25T19:25:20.564 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.562+0000 7f7fffeed640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff8139870 con 0x7f7ff81430c0 2024-03-25T19:25:20.564 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.562+0000 7f7fffeed640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff81d51f0 con 0x7f7ff80a98f0 2024-03-25T19:25:20.564 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.562+0000 7f7fffeed640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff8138830 con 0x7f7ff813f710 2024-03-25T19:25:20.565 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.562+0000 7f7fffeed640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:20.565 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.562+0000 7f7fffeed640 10 monclient(hunting): authenticate will time out at 6377.834170s 2024-03-25T19:25:20.571 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.572+0000 7f7ffe48a640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f7ff81430c0 0x7f7ff81e1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:44994/0 (socket says 172.21.15.83:44994) 2024-03-25T19:25:20.571 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.572+0000 7f7ffe48a640 1 -- 172.21.15.83:0/499144132 learned_addr learned my addr 172.21.15.83:0/499144132 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:25:20.572 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.572+0000 7f7fee7fc640 10 client.?.objecter ms_handle_connect 0x7f7ff81430c0 2024-03-25T19:25:20.572 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.572+0000 7f7fee7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:20.581 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.581+0000 7f7fee7fc640 10 client.?.objecter ms_handle_connect 0x7f7ff80a98f0 2024-03-25T19:25:20.581 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.581+0000 7f7fee7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:20.581 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.582+0000 7f7fee7fc640 10 client.?.objecter ms_handle_connect 0x7f7ff813f710 2024-03-25T19:25:20.582 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.582+0000 7f7fee7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:20.583 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.583+0000 7f7fee7fc640 1 -- 172.21.15.83:0/499144132 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1369681622 0 0) 0x7f7ff8139870 con 0x7f7ff81430c0 2024-03-25T19:25:20.583 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.583+0000 7f7fee7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:25:20.583 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.583+0000 7f7fee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:20.583 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.583+0000 7f7fee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:20.583 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.583+0000 7f7fee7fc640 10 monclient(hunting): my global_id is 4473 2024-03-25T19:25:20.583 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.583+0000 7f7fee7fc640 1 -- 172.21.15.83:0/499144132 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fdc004eb0 con 0x7f7ff81430c0 2024-03-25T19:25:20.590 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.590+0000 7f7fee7fc640 1 -- 172.21.15.83:0/499144132 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 923945985 0 0) 0x7f7ff81d51f0 con 0x7f7ff80a98f0 2024-03-25T19:25:20.590 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.590+0000 7f7fee7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:25:20.590 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.590+0000 7f7fee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:20.590 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.590+0000 7f7fee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:20.590 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.590+0000 7f7fee7fc640 10 monclient(hunting): my global_id is 4483 2024-03-25T19:25:20.590 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.590+0000 7f7fee7fc640 1 -- 172.21.15.83:0/499144132 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ff8139870 con 0x7f7ff80a98f0 2024-03-25T19:25:20.595 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.596+0000 7f7fee7fc640 1 -- 172.21.15.83:0/499144132 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2670468461 0 0) 0x7f7ff8138830 con 0x7f7ff813f710 2024-03-25T19:25:20.595 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.596+0000 7f7fee7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:25:20.595 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.596+0000 7f7fee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:20.596 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.596+0000 7f7fee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:20.596 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.596+0000 7f7fee7fc640 10 monclient(hunting): my global_id is 4490 2024-03-25T19:25:20.596 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.596+0000 7f7fee7fc640 1 -- 172.21.15.83:0/499144132 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ff81d51f0 con 0x7f7ff813f710 2024-03-25T19:25:20.598 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.598+0000 7f7fee7fc640 1 -- 172.21.15.83:0/499144132 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3360639774 0 0) 0x7f7fdc004eb0 con 0x7f7ff81430c0 2024-03-25T19:25:20.598 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.598+0000 7f7fee7fc640 1 -- 172.21.15.83:0/499144132 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ff8138830 con 0x7f7ff81430c0 2024-03-25T19:25:20.598 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.598+0000 7f7fee7fc640 1 -- 172.21.15.83:0/499144132 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f7ff0001c90 con 0x7f7ff81430c0 2024-03-25T19:25:20.598 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.598+0000 7f7fee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:20.598 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.598+0000 7f7fee7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:25:20.599 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.598+0000 7f7fee7fc640 10 monclient(hunting): dump: 2024-03-25T19:25:20.599 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:25:20.599 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:25:20.599 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:20.599 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:20.599 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:20.599 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:25:20.599 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:25:20.599 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:25:20.599 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:25:20.599 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:20.599 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.598+0000 7f7fee7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:25:20.607 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.608+0000 7f7fee7fc640 1 -- 172.21.15.83:0/499144132 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4074496951 0 0) 0x7f7ff8139870 con 0x7f7ff80a98f0 2024-03-25T19:25:20.608 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.608+0000 7f7fee7fc640 1 -- 172.21.15.83:0/499144132 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7fdc004eb0 con 0x7f7ff80a98f0 2024-03-25T19:25:20.608 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.608+0000 7f7fee7fc640 1 -- 172.21.15.83:0/499144132 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f7ff4001b00 con 0x7f7ff80a98f0 2024-03-25T19:25:20.608 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.608+0000 7f7fee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:20.608 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.608+0000 7f7fee7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:25:20.608 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.608+0000 7f7fee7fc640 10 monclient(hunting): dump: 2024-03-25T19:25:20.608 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:25:20.608 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:25:20.608 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:20.608 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:20.608 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:20.608 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:25:20.608 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:25:20.608 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:25:20.608 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:25:20.608 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:20.610 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.610+0000 7f7fee7fc640 1 -- 172.21.15.83:0/499144132 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2906101296 0 0) 0x7f7ff8138830 con 0x7f7ff81430c0 2024-03-25T19:25:20.610 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.610+0000 7f7fee7fc640 1 -- 172.21.15.83:0/499144132 >> v1:172.21.15.83:6790/0 conn(0x7f7ff813f710 legacy=0x7f7ff81d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:20.610 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.610+0000 7f7fee7fc640 1 -- 172.21.15.83:0/499144132 >> v1:172.21.15.103:6789/0 conn(0x7f7ff80a98f0 legacy=0x7f7ff8138120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:20.610 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.610+0000 7f7fee7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:25:20.610 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.610+0000 7f7fee7fc640 1 monclient: found mon.a 2024-03-25T19:25:20.610 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.610+0000 7f7fee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:20.610 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.610+0000 7f7fee7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:25:20.610 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.610+0000 7f7fee7fc640 1 -- 172.21.15.83:0/499144132 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff81d77a0 con 0x7f7ff81430c0 2024-03-25T19:25:20.610 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.610+0000 7f7fee7fc640 10 monclient: _finish_auth 0 2024-03-25T19:25:20.611 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.610+0000 7f7fee7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:25:20.611 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.610+0000 7f7fee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:20.611 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.610+0000 7f7fffeed640 5 monclient: authenticate success, global_id 4473 2024-03-25T19:25:20.611 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.610+0000 7f7fffeed640 10 monclient: _renew_subs 2024-03-25T19:25:20.611 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.610+0000 7f7fffeed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:25:20.611 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.610+0000 7f7fffeed640 1 -- 172.21.15.83:0/499144132 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ff81d83d0 con 0x7f7ff81430c0 2024-03-25T19:25:20.611 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.610+0000 7f7fffeed640 10 client.4473.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:25:20.611 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.610+0000 7f7fffeed640 10 monclient: _renew_subs 2024-03-25T19:25:20.611 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.610+0000 7f7fffeed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:25:20.611 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.610+0000 7f7fffeed640 1 -- 172.21.15.83:0/499144132 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ff8138c80 con 0x7f7ff81430c0 2024-03-25T19:25:20.611 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.610+0000 7f7fffeed640 1 librados: init done 2024-03-25T19:25:20.612 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.611+0000 7f7fffeed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:20.612 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.611+0000 7f7fffeed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:20.612 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.611+0000 7f7fffeed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:25:20.612 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.611+0000 7f7fffeed640 1 -- 172.21.15.83:0/499144132 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ff80b3e80 con 0x7f7ff81430c0 2024-03-25T19:25:20.614 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.614+0000 7f7fee7fc640 1 -- 172.21.15.83:0/499144132 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ff0003990 con 0x7f7ff81430c0 2024-03-25T19:25:20.614 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.614+0000 7f7fee7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:20.614 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.614+0000 7f7feffff640 4 set_mon_vals no callback set 2024-03-25T19:25:20.636 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.636+0000 7f7ffe48a640 1 -- 172.21.15.83:0/499144132 >> v1:172.21.15.83:6789/0 conn(0x7f7ff81430c0 legacy=0x7f7ff81e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-25T19:25:20.636 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.636+0000 7f7ffe48a640 1 -- 172.21.15.83:0/499144132 >> v1:172.21.15.83:6789/0 conn(0x7f7ff81430c0 legacy=0x7f7ff81e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:25:20.636 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.636+0000 7f7fee7fc640 1 -- 172.21.15.83:0/499144132 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f7ff0003ca0 con 0x7f7ff81430c0 2024-03-25T19:25:20.636 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.636+0000 7f7ffe48a640 1 --1- 172.21.15.83:0/499144132 >> v1:172.21.15.83:6789/0 conn(0x7f7ff81430c0 0x7f7ff81e1e30 :-1 s=OPENED pgs=38803 cs=1 l=1).handle_message read tag failed 2024-03-25T19:25:20.636 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.636+0000 7f7fee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:20.637 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.636+0000 7f7ffe48a640 1 --1- 172.21.15.83:0/499144132 >> v1:172.21.15.83:6789/0 conn(0x7f7ff81430c0 0x7f7ff81e1e30 :-1 s=OPENED pgs=38803 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:25:20.637 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.636+0000 7f7fee7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:25:20.637 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.636+0000 7f7fee7fc640 10 monclient: dump: 2024-03-25T19:25:20.637 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:25:20.637 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:25:20.637 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:20.637 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:20.637 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:20.637 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:25:20.637 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:25:20.637 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:25:20.637 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:25:20.637 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:20.637 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.636+0000 7f7fee7fc640 10 monclient: ms_handle_reset current mon v1:172.21.15.83:6789/0 2024-03-25T19:25:20.637 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.636+0000 7f7fee7fc640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:20.637 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.636+0000 7f7fee7fc640 1 -- 172.21.15.83:0/499144132 >> v1:172.21.15.83:6789/0 conn(0x7f7ff81430c0 legacy=0x7f7ff81e1e30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:25:20.638 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.636+0000 7f7fee7fc640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:25:20.638 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.636+0000 7f7fee7fc640 10 monclient(hunting): picked mon.b con 0x7f7ff80a98f0 addr v1:172.21.15.103:6789/0 2024-03-25T19:25:20.638 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.636+0000 7f7fee7fc640 10 monclient(hunting): picked mon.c con 0x7f7ff813f710 addr v1:172.21.15.83:6790/0 2024-03-25T19:25:20.638 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.636+0000 7f7fee7fc640 10 monclient(hunting): picked mon.a con 0x7f7fdc00dc90 addr v1:172.21.15.83:6789/0 2024-03-25T19:25:20.638 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.636+0000 7f7fee7fc640 1 -- 172.21.15.83:0/499144132 --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7ff8138830 con 0x7f7fdc00dc90 2024-03-25T19:25:20.638 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.636+0000 7f7fee7fc640 1 -- 172.21.15.83:0/499144132 --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7ff8139870 con 0x7f7ff80a98f0 2024-03-25T19:25:20.638 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.636+0000 7f7fee7fc640 1 -- 172.21.15.83:0/499144132 --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7fdc013dc0 con 0x7f7ff813f710 2024-03-25T19:25:20.638 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.636+0000 7f7fee7fc640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:20.640 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.640+0000 7f7fee7fc640 10 client.4473.objecter ms_handle_connect 0x7f7ff80a98f0 2024-03-25T19:25:20.640 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.640+0000 7f7fee7fc640 10 client.4473.objecter resend_mon_ops 2024-03-25T19:25:20.651 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.651+0000 7f7fee7fc640 1 -- 172.21.15.83:0/499144132 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1408630700 0 0) 0x7f7ff8139870 con 0x7f7ff80a98f0 2024-03-25T19:25:20.651 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.651+0000 7f7fee7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:25:20.651 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.651+0000 7f7fee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:20.651 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.651+0000 7f7fee7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-25T19:25:20.652 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.651+0000 7f7fee7fc640 1 -- 172.21.15.83:0/499144132 --> v1:172.21.15.103:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f7fdc012d40 con 0x7f7ff80a98f0 2024-03-25T19:25:20.654 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.654+0000 7f7fee7fc640 10 client.4473.objecter ms_handle_connect 0x7f7ff813f710 2024-03-25T19:25:20.654 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.655+0000 7f7fee7fc640 10 client.4473.objecter resend_mon_ops 2024-03-25T19:25:20.662 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.662+0000 7f7fee7fc640 10 client.4473.objecter ms_handle_connect 0x7f7fdc00dc90 2024-03-25T19:25:20.662 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.662+0000 7f7fee7fc640 10 client.4473.objecter resend_mon_ops 2024-03-25T19:25:20.664 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.664+0000 7f7fee7fc640 1 -- 172.21.15.83:0/499144132 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2545401931 0 0) 0x7f7fdc013dc0 con 0x7f7ff813f710 2024-03-25T19:25:20.664 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.664+0000 7f7fee7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:25:20.664 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.664+0000 7f7fee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:20.664 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.664+0000 7f7fee7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-25T19:25:20.664 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.664+0000 7f7fee7fc640 1 -- 172.21.15.83:0/499144132 --> v1:172.21.15.83:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f7ff8139870 con 0x7f7ff813f710 2024-03-25T19:25:20.665 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.665+0000 7f7fee7fc640 1 -- 172.21.15.83:0/499144132 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 1546256119 0 0) 0x7f7fdc012d40 con 0x7f7ff80a98f0 2024-03-25T19:25:20.666 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.665+0000 7f7fee7fc640 1 -- 172.21.15.83:0/499144132 >> v1:172.21.15.83:6790/0 conn(0x7f7ff813f710 legacy=0x7f7fdc00c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:20.666 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.665+0000 7f7fee7fc640 1 -- 172.21.15.83:0/499144132 >> v1:172.21.15.83:6789/0 conn(0x7f7fdc00dc90 legacy=0x7f7fdc010120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:20.666 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.665+0000 7f7fee7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:25:20.666 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.665+0000 7f7fee7fc640 1 monclient: found mon.b 2024-03-25T19:25:20.666 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.665+0000 7f7fee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:20.666 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.665+0000 7f7fee7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:25:20.666 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.665+0000 7f7fee7fc640 1 -- 172.21.15.83:0/499144132 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f7fdc0151a0 con 0x7f7ff80a98f0 2024-03-25T19:25:20.666 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.665+0000 7f7fee7fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:20.666 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.665+0000 7f7fee7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:25:20.666 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.665+0000 7f7fee7fc640 1 -- 172.21.15.83:0/499144132 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fdc013dc0 con 0x7f7ff80a98f0 2024-03-25T19:25:20.666 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.665+0000 7f7fee7fc640 10 monclient: _finish_auth 0 2024-03-25T19:25:20.666 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.665+0000 7f7fee7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:25:20.667 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.665+0000 7f7fee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:20.667 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.665+0000 7f7fee7fc640 1 -- 172.21.15.83:0/499144132 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f7ff4004980 con 0x7f7ff80a98f0 2024-03-25T19:25:20.667 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.665+0000 7f7fee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:20.667 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.665+0000 7f7fee7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:25:20.667 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.665+0000 7f7fee7fc640 10 monclient: dump: 2024-03-25T19:25:20.667 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:25:20.667 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:25:20.667 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:20.667 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:20.667 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:20.667 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:25:20.667 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:25:20.667 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:25:20.667 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:25:20.667 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:20.967 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.968+0000 7f7fee7fc640 1 -- 172.21.15.83:0/499144132 <== mon.1 v1:172.21.15.103:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ff4004ae0 con 0x7f7ff80a98f0 2024-03-25T19:25:20.968 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.968+0000 7f7fee7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:20.968 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.968+0000 7f7fef7fe640 4 set_mon_vals no callback set 2024-03-25T19:25:20.968 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.969+0000 7f7fee7fc640 1 -- 172.21.15.83:0/499144132 <== mon.1 v1:172.21.15.103:6789/0 5 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7f7ff401aac0 con 0x7f7ff80a98f0 2024-03-25T19:25:20.969 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.969+0000 7f7ffd488640 10 monclient: get_auth_request con 0x7f7ff81430c0 auth_method 0 2024-03-25T19:25:20.969 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.970+0000 7f7fee7fc640 10 client.4473.objecter ms_handle_connect 0x7f7ff81430c0 2024-03-25T19:25:20.974 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.974+0000 7f7fee7fc640 1 -- 172.21.15.83:0/499144132 <== mon.1 v1:172.21.15.103:6789/0 6 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7f7ff4054760 con 0x7f7ff80a98f0 2024-03-25T19:25:20.974 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.974+0000 7f7fee7fc640 10 client.4473.objecter ms_dispatch 0x7f7ff8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:25:20.974 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.974+0000 7f7fee7fc640 3 client.4473.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:25:20.974 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.974+0000 7f7fee7fc640 3 client.4473.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:25:20.974 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.974+0000 7f7fee7fc640 20 client.4473.objecter dump_active .. 0 homeless 2024-03-25T19:25:20.976 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.976+0000 7f7fee7fc640 1 -- 172.21.15.83:0/499144132 <== mon.1 v1:172.21.15.103:6789/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7ff40500a0 con 0x7f7ff80a98f0 2024-03-25T19:25:20.976 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.976+0000 7f7fee7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:20.976 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:20.976+0000 7f7fee7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:25:21.220 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:21.220+0000 7f7fffeed640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:21.220 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:21.220+0000 7f7fffeed640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:21.221 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:21.220+0000 7f7fffeed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:25:21.221 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:21.220+0000 7f7fffeed640 1 -- 172.21.15.83:0/499144132 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7ff80627b0 con 0x7f7ff80a98f0 2024-03-25T19:25:21.251 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:21.252+0000 7f7fee7fc640 1 -- 172.21.15.83:0/499144132 <== mon.1 v1:172.21.15.103:6789/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7f7ff40521a0 con 0x7f7ff80a98f0 2024-03-25T19:25:21.251 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:21.252+0000 7f7fee7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:21.251 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:21.252+0000 7f7fee7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:25:21.252 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:25:21.252 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:25:21.254 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:21.254+0000 7f7fffeed640 10 librados: watch_flush enter 2024-03-25T19:25:21.254 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:21.254+0000 7f7fffeed640 10 librados: watch_flush exit 2024-03-25T19:25:21.255 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:21.254+0000 7f7fffeed640 20 client.4473.objecter shutdown clearing up homeless session... 2024-03-25T19:25:21.255 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:21.254+0000 7f7fffeed640 10 client.4473.objecter successfully canceled tick 2024-03-25T19:25:21.255 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:21.254+0000 7f7fffeed640 1 -- 172.21.15.83:0/499144132 >> v1:172.21.15.103:6800/345072327 conn(0x7f7ff81430c0 legacy=0x7f7fdc07ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:21.255 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:21.254+0000 7f7fffeed640 10 monclient: shutdown 2024-03-25T19:25:21.255 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:21.254+0000 7f7fffeed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:21.255 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:21.254+0000 7f7fffeed640 1 -- 172.21.15.83:0/499144132 >> v1:172.21.15.103:6789/0 conn(0x7f7ff80a98f0 legacy=0x7f7fdc006740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:21.255 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:21.254+0000 7f7fffeed640 1 -- 172.21.15.83:0/499144132 shutdown_connections 2024-03-25T19:25:21.255 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:21.254+0000 7f7fffeed640 1 -- 172.21.15.83:0/499144132 >> 172.21.15.83:0/499144132 conn(0x7f7ff80b7ea0 msgr2=0x7f7ff8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:21.255 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:21.254+0000 7f7fffeed640 1 -- 172.21.15.83:0/499144132 shutdown_connections 2024-03-25T19:25:21.255 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:21.254+0000 7f7fffeed640 1 -- 172.21.15.83:0/499144132 wait complete. 2024-03-25T19:25:21.255 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:21.254+0000 7f7fffeed640 1 librados: shutdown 2024-03-25T19:25:21.265 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:25:23.189 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:06:05.116 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:23.209 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:06:05.136 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:23.248 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:06:05.158 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:23.248 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:06:05.176 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:23.252 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:06:05.179 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:23.256 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:06:05.184 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:23.256 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:06:05.166 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:23.283 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:06:05.210 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:23.284 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:06:05.194 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:28.071 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:25:28.188 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.188+0000 7fa6dc7d0640 10 monclient: get_monmap_and_config 2024-03-25T19:25:28.188 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.188+0000 7fa6dc7d0640 10 monclient: build_initial_monmap 2024-03-25T19:25:28.188 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.188+0000 7fa6dc7d0640 10 monclient: monmap: 2024-03-25T19:25:28.188 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:25:28.188 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:25:28.188 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:25:28.189452+0000 2024-03-25T19:25:28.188 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:25:28.189452+0000 2024-03-25T19:25:28.188 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:28.188 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:25:28.189 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:25:28.189 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:25:28.189 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:25:28.189 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:28.190 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.189+0000 7fa6dc7d0640 1 Processor -- start 2024-03-25T19:25:28.190 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.189+0000 7fa6dc7d0640 1 -- start start 2024-03-25T19:25:28.190 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.189+0000 7fa6dc7d0640 10 monclient: init 2024-03-25T19:25:28.190 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.189+0000 7fa6dc7d0640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:28.190 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.189+0000 7fa6dc7d0640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:25:28.190 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.189+0000 7fa6dc7d0640 10 monclient(hunting): picked mon.noname-c con 0x7fa6d40a98f0 addr v1:172.21.15.103:6789/0 2024-03-25T19:25:28.190 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.190+0000 7fa6dc7d0640 10 monclient(hunting): picked mon.noname-b con 0x7fa6d413f710 addr v1:172.21.15.83:6790/0 2024-03-25T19:25:28.190 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.190+0000 7fa6dc7d0640 10 monclient(hunting): picked mon.noname-a con 0x7fa6d41430c0 addr v1:172.21.15.83:6789/0 2024-03-25T19:25:28.190 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.190+0000 7fa6dc7d0640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6d4142520 con 0x7fa6d41430c0 2024-03-25T19:25:28.190 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.190+0000 7fa6dc7d0640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6d40b28a0 con 0x7fa6d40a98f0 2024-03-25T19:25:28.190 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.190+0000 7fa6dc7d0640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6d40b2a30 con 0x7fa6d413f710 2024-03-25T19:25:28.190 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.190+0000 7fa6dc7d0640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:28.191 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.190+0000 7fa6dc7d0640 10 monclient(hunting): authenticate will time out at 6385.461841s 2024-03-25T19:25:28.192 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.192+0000 7fa6da56c640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fa6d40a98f0 0x7fa6d40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:40752/0 (socket says 172.21.15.83:40752) 2024-03-25T19:25:28.192 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.192+0000 7fa6da56c640 1 -- 172.21.15.83:0/1392014154 learned_addr learned my addr 172.21.15.83:0/1392014154 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:25:28.206 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.206+0000 7fa6d956a640 1 -- 172.21.15.83:0/1392014154 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3981613742 0 0) 0x7fa6d40b28a0 con 0x7fa6d40a98f0 2024-03-25T19:25:28.206 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.206+0000 7fa6d956a640 10 monclient(hunting): _negotiate 2024-03-25T19:25:28.206 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.206+0000 7fa6d956a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:28.206 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.206+0000 7fa6d956a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:28.206 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.206+0000 7fa6d956a640 10 monclient(hunting): my global_id is 4486 2024-03-25T19:25:28.206 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.206+0000 7fa6d956a640 1 -- 172.21.15.83:0/1392014154 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6c0003520 con 0x7fa6d40a98f0 2024-03-25T19:25:28.217 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.217+0000 7fa6d956a640 1 -- 172.21.15.83:0/1392014154 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1749017659 0 0) 0x7fa6d4142520 con 0x7fa6d41430c0 2024-03-25T19:25:28.217 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.217+0000 7fa6d956a640 10 monclient(hunting): _negotiate 2024-03-25T19:25:28.217 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.217+0000 7fa6d956a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:28.217 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.217+0000 7fa6d956a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:28.217 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.217+0000 7fa6d956a640 10 monclient(hunting): my global_id is 4476 2024-03-25T19:25:28.218 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.218+0000 7fa6d956a640 1 -- 172.21.15.83:0/1392014154 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6d40b28a0 con 0x7fa6d41430c0 2024-03-25T19:25:28.226 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.226+0000 7fa6d956a640 1 -- 172.21.15.83:0/1392014154 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1697343634 0 0) 0x7fa6c0003520 con 0x7fa6d40a98f0 2024-03-25T19:25:28.226 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.226+0000 7fa6d956a640 1 -- 172.21.15.83:0/1392014154 >> v1:172.21.15.83:6790/0 conn(0x7fa6d413f710 legacy=0x7fa6d41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:28.226 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.226+0000 7fa6d956a640 1 -- 172.21.15.83:0/1392014154 >> v1:172.21.15.83:6789/0 conn(0x7fa6d41430c0 legacy=0x7fa6d414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:28.226 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.226+0000 7fa6d956a640 10 monclient: _finish_hunting 0 2024-03-25T19:25:28.226 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.226+0000 7fa6d956a640 1 monclient: found mon.noname-c 2024-03-25T19:25:28.226 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.226+0000 7fa6d956a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:28.226 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.226+0000 7fa6d956a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.103:6789/0 2024-03-25T19:25:28.226 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.226+0000 7fa6d956a640 1 -- 172.21.15.83:0/1392014154 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d40b44c0 con 0x7fa6d40a98f0 2024-03-25T19:25:28.226 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.226+0000 7fa6d956a640 10 monclient: _finish_auth 0 2024-03-25T19:25:28.226 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.226+0000 7fa6d956a640 10 monclient: _check_auth_tickets 2024-03-25T19:25:28.226 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.226+0000 7fa6d956a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:28.226 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.226+0000 7fa6d956a640 1 -- 172.21.15.83:0/1392014154 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fa6d0001690 con 0x7fa6d40a98f0 2024-03-25T19:25:28.227 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.226+0000 7fa6d956a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:28.227 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.226+0000 7fa6d956a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:25:28.227 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.226+0000 7fa6d956a640 10 monclient: dump: 2024-03-25T19:25:28.227 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:25:28.227 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:25:28.227 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:28.227 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:28.227 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:28.227 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:25:28.227 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:25:28.227 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:25:28.227 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:25:28.227 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:28.227 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.226+0000 7fa6dc7d0640 5 monclient: authenticate success, global_id 4486 2024-03-25T19:25:28.228 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.226+0000 7fa6dc7d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:25:28.235 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.235+0000 7fa6d956a640 1 -- 172.21.15.83:0/1392014154 <== mon.1 v1:172.21.15.103:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa6d00017f0 con 0x7fa6d40a98f0 2024-03-25T19:25:28.235 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.235+0000 7fa6d956a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:28.235 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.235+0000 7fa6d956a640 1 -- 172.21.15.83:0/1392014154 <== mon.1 v1:172.21.15.103:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fa6d0001c00 con 0x7fa6d40a98f0 2024-03-25T19:25:28.235 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.235+0000 7fa6d956a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:28.235 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.235+0000 7fa6d956a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:25:28.235 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.235+0000 7fa6d956a640 10 monclient: dump: 2024-03-25T19:25:28.235 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:25:28.235 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:25:28.235 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:28.235 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:28.235 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:28.235 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:25:28.236 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:25:28.236 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:25:28.236 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:25:28.236 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:28.236 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.235+0000 7fa6dc7d0640 10 monclient: get_monmap_and_config success 2024-03-25T19:25:28.236 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.235+0000 7fa6dc7d0640 4 set_mon_vals no callback set 2024-03-25T19:25:28.239 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.236+0000 7fa6dc7d0640 10 monclient: shutdown 2024-03-25T19:25:28.239 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.236+0000 7fa6dc7d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:28.239 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.236+0000 7fa6dc7d0640 1 -- 172.21.15.83:0/1392014154 >> v1:172.21.15.103:6789/0 conn(0x7fa6d40a98f0 legacy=0x7fa6d40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:28.239 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.236+0000 7fa6dc7d0640 1 -- 172.21.15.83:0/1392014154 shutdown_connections 2024-03-25T19:25:28.239 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.236+0000 7fa6dc7d0640 1 -- 172.21.15.83:0/1392014154 >> 172.21.15.83:0/1392014154 conn(0x7fa6d40b7ea0 msgr2=0x7fa6d40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:28.239 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.236+0000 7fa6dc7d0640 1 -- 172.21.15.83:0/1392014154 shutdown_connections 2024-03-25T19:25:28.239 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.236+0000 7fa6dc7d0640 1 -- 172.21.15.83:0/1392014154 wait complete. 2024-03-25T19:25:28.239 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.236+0000 7fa6dc7d0640 10 monclient: build_initial_monmap 2024-03-25T19:25:28.239 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.237+0000 7fa6dc7d0640 10 monclient: monmap: 2024-03-25T19:25:28.239 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:25:28.239 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:25:28.239 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:25:28.239 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:25:28.239 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:28.239 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:25:28.240 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:25:28.240 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:25:28.240 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:25:28.240 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:28.240 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.237+0000 7fa6dc7d0640 1 librados: starting msgr at 2024-03-25T19:25:28.240 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.237+0000 7fa6dc7d0640 1 librados: starting objecter 2024-03-25T19:25:28.240 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.237+0000 7fa6dc7d0640 1 Processor -- start 2024-03-25T19:25:28.240 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.237+0000 7fa6dc7d0640 1 -- start start 2024-03-25T19:25:28.240 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.237+0000 7fa6dc7d0640 1 librados: setting wanted keys 2024-03-25T19:25:28.240 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.237+0000 7fa6dc7d0640 1 librados: calling monclient init 2024-03-25T19:25:28.240 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.237+0000 7fa6dc7d0640 10 monclient: init 2024-03-25T19:25:28.240 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.237+0000 7fa6dc7d0640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:28.240 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.238+0000 7fa6dc7d0640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:25:28.240 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.238+0000 7fa6dc7d0640 10 monclient(hunting): picked mon.noname-b con 0x7fa6d40a98f0 addr v1:172.21.15.103:6789/0 2024-03-25T19:25:28.240 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.238+0000 7fa6dc7d0640 10 monclient(hunting): picked mon.noname-c con 0x7fa6d413f710 addr v1:172.21.15.83:6790/0 2024-03-25T19:25:28.240 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.238+0000 7fa6dc7d0640 10 monclient(hunting): picked mon.noname-a con 0x7fa6d41430c0 addr v1:172.21.15.83:6789/0 2024-03-25T19:25:28.241 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.238+0000 7fa6dc7d0640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6d4138250 con 0x7fa6d41430c0 2024-03-25T19:25:28.241 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.238+0000 7fa6dc7d0640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6d4137d50 con 0x7fa6d40a98f0 2024-03-25T19:25:28.241 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.238+0000 7fa6dc7d0640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6d4137ee0 con 0x7fa6d413f710 2024-03-25T19:25:28.241 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.238+0000 7fa6dc7d0640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:28.241 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.238+0000 7fa6dc7d0640 10 monclient(hunting): authenticate will time out at 6385.509991s 2024-03-25T19:25:28.242 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.243+0000 7fa6dad6d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fa6d41430c0 0x7fa6d41d7df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49664/0 (socket says 172.21.15.83:49664) 2024-03-25T19:25:28.242 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.243+0000 7fa6dad6d640 1 -- 172.21.15.83:0/2735269963 learned_addr learned my addr 172.21.15.83:0/2735269963 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:25:28.245 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.245+0000 7fa6caffd640 10 client.?.objecter ms_handle_connect 0x7fa6d41430c0 2024-03-25T19:25:28.245 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.245+0000 7fa6caffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:28.251 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.251+0000 7fa6caffd640 10 client.?.objecter ms_handle_connect 0x7fa6d413f710 2024-03-25T19:25:28.251 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.251+0000 7fa6caffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:28.258 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.259+0000 7fa6caffd640 10 client.?.objecter ms_handle_connect 0x7fa6d40a98f0 2024-03-25T19:25:28.259 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.259+0000 7fa6caffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:28.265 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.265+0000 7fa6caffd640 1 -- 172.21.15.83:0/2735269963 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1621863639 0 0) 0x7fa6d4138250 con 0x7fa6d41430c0 2024-03-25T19:25:28.265 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.265+0000 7fa6caffd640 10 monclient(hunting): _negotiate 2024-03-25T19:25:28.265 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.265+0000 7fa6caffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:28.265 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.265+0000 7fa6caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:28.265 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.265+0000 7fa6caffd640 10 monclient(hunting): my global_id is 4479 2024-03-25T19:25:28.265 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.265+0000 7fa6caffd640 1 -- 172.21.15.83:0/2735269963 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6b8004eb0 con 0x7fa6d41430c0 2024-03-25T19:25:28.269 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.269+0000 7fa6caffd640 1 -- 172.21.15.83:0/2735269963 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 575300339 0 0) 0x7fa6d4137d50 con 0x7fa6d40a98f0 2024-03-25T19:25:28.269 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.270+0000 7fa6caffd640 10 monclient(hunting): _negotiate 2024-03-25T19:25:28.269 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.270+0000 7fa6caffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:28.269 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.270+0000 7fa6caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:28.270 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.270+0000 7fa6caffd640 10 monclient(hunting): my global_id is 4489 2024-03-25T19:25:28.270 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.270+0000 7fa6caffd640 1 -- 172.21.15.83:0/2735269963 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6d4138250 con 0x7fa6d40a98f0 2024-03-25T19:25:28.270 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.271+0000 7fa6caffd640 1 -- 172.21.15.83:0/2735269963 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1349241494 0 0) 0x7fa6d4137ee0 con 0x7fa6d413f710 2024-03-25T19:25:28.270 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.271+0000 7fa6caffd640 10 monclient(hunting): _negotiate 2024-03-25T19:25:28.270 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.271+0000 7fa6caffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:28.271 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.271+0000 7fa6caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:28.271 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.271+0000 7fa6caffd640 10 monclient(hunting): my global_id is 4496 2024-03-25T19:25:28.271 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.271+0000 7fa6caffd640 1 -- 172.21.15.83:0/2735269963 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6d4137d50 con 0x7fa6d413f710 2024-03-25T19:25:28.275 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.275+0000 7fa6caffd640 1 -- 172.21.15.83:0/2735269963 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2387779506 0 0) 0x7fa6b8004eb0 con 0x7fa6d41430c0 2024-03-25T19:25:28.275 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.275+0000 7fa6caffd640 1 -- 172.21.15.83:0/2735269963 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6d4137ee0 con 0x7fa6d41430c0 2024-03-25T19:25:28.283 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.283+0000 7fa6caffd640 1 -- 172.21.15.83:0/2735269963 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fa6cc001da0 con 0x7fa6d41430c0 2024-03-25T19:25:28.283 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.283+0000 7fa6caffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:28.283 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.283+0000 7fa6caffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:25:28.283 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.283+0000 7fa6caffd640 10 monclient(hunting): dump: 2024-03-25T19:25:28.283 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:25:28.283 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:25:28.283 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:28.283 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:28.283 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:28.283 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:25:28.283 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:25:28.283 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:25:28.284 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:25:28.284 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:28.284 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.283+0000 7fa6caffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:25:28.284 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.283+0000 7fa6caffd640 1 -- 172.21.15.83:0/2735269963 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3468011243 0 0) 0x7fa6d4138250 con 0x7fa6d40a98f0 2024-03-25T19:25:28.284 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.283+0000 7fa6caffd640 1 -- 172.21.15.83:0/2735269963 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6b8004eb0 con 0x7fa6d40a98f0 2024-03-25T19:25:28.284 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.283+0000 7fa6caffd640 1 -- 172.21.15.83:0/2735269963 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fa6d0001b00 con 0x7fa6d40a98f0 2024-03-25T19:25:28.285 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.283+0000 7fa6caffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:28.285 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.284+0000 7fa6caffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:25:28.285 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.284+0000 7fa6caffd640 10 monclient(hunting): dump: 2024-03-25T19:25:28.285 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:25:28.285 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:25:28.285 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:28.285 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:28.285 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:28.285 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:25:28.285 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:25:28.285 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:25:28.285 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:25:28.285 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:28.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.290+0000 7fa6caffd640 1 -- 172.21.15.83:0/2735269963 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1278766561 0 0) 0x7fa6d4137ee0 con 0x7fa6d41430c0 2024-03-25T19:25:28.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.290+0000 7fa6caffd640 1 -- 172.21.15.83:0/2735269963 >> v1:172.21.15.83:6790/0 conn(0x7fa6d413f710 legacy=0x7fa6d41d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:28.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.290+0000 7fa6caffd640 1 -- 172.21.15.83:0/2735269963 >> v1:172.21.15.103:6789/0 conn(0x7fa6d40a98f0 legacy=0x7fa6d4137640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:28.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.290+0000 7fa6caffd640 10 monclient: _finish_hunting 0 2024-03-25T19:25:28.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.290+0000 7fa6caffd640 1 monclient: found mon.a 2024-03-25T19:25:28.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.291+0000 7fa6caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:28.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.291+0000 7fa6caffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:25:28.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.291+0000 7fa6caffd640 1 -- 172.21.15.83:0/2735269963 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d40b7730 con 0x7fa6d41430c0 2024-03-25T19:25:28.292 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.291+0000 7fa6caffd640 10 monclient: _finish_auth 0 2024-03-25T19:25:28.292 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.291+0000 7fa6caffd640 10 monclient: _check_auth_tickets 2024-03-25T19:25:28.292 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.291+0000 7fa6caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:28.292 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.291+0000 7fa6dc7d0640 5 monclient: authenticate success, global_id 4479 2024-03-25T19:25:28.292 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.291+0000 7fa6dc7d0640 10 monclient: _renew_subs 2024-03-25T19:25:28.292 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.291+0000 7fa6dc7d0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:25:28.292 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.291+0000 7fa6dc7d0640 1 -- 172.21.15.83:0/2735269963 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6d40b7920 con 0x7fa6d41430c0 2024-03-25T19:25:28.292 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.291+0000 7fa6dc7d0640 10 client.4479.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:25:28.292 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.291+0000 7fa6dc7d0640 10 monclient: _renew_subs 2024-03-25T19:25:28.292 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.291+0000 7fa6dc7d0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:25:28.292 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.291+0000 7fa6dc7d0640 1 -- 172.21.15.83:0/2735269963 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6d41e33d0 con 0x7fa6d41430c0 2024-03-25T19:25:28.292 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.291+0000 7fa6dc7d0640 1 librados: init done 2024-03-25T19:25:28.293 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.292+0000 7fa6dc7d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:28.293 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.292+0000 7fa6dc7d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:28.293 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.292+0000 7fa6dc7d0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:25:28.293 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.292+0000 7fa6dc7d0640 1 -- 172.21.15.83:0/2735269963 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6d40627b0 con 0x7fa6d41430c0 2024-03-25T19:25:28.623 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.623+0000 7fa6caffd640 1 -- 172.21.15.83:0/2735269963 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa6cc001f00 con 0x7fa6d41430c0 2024-03-25T19:25:28.623 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.623+0000 7fa6caffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:28.623 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.623+0000 7fa6caffd640 1 -- 172.21.15.83:0/2735269963 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fa6cc003de0 con 0x7fa6d41430c0 2024-03-25T19:25:28.623 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.623+0000 7fa6caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:28.624 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.623+0000 7fa6caffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:25:28.624 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.623+0000 7fa6caffd640 10 monclient: dump: 2024-03-25T19:25:28.624 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:25:28.624 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:25:28.624 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:28.624 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:28.624 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:28.625 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:25:28.625 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:25:28.625 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:25:28.625 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:25:28.625 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:28.626 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.623+0000 7fa6d8d69640 4 set_mon_vals no callback set 2024-03-25T19:25:28.627 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.624+0000 7fa6caffd640 1 -- 172.21.15.83:0/2735269963 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7fa6cc004000 con 0x7fa6d41430c0 2024-03-25T19:25:28.627 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.625+0000 7fa6da56c640 10 monclient: get_auth_request con 0x7fa6b806df60 auth_method 0 2024-03-25T19:25:28.627 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.625+0000 7fa6caffd640 10 client.4479.objecter ms_handle_connect 0x7fa6b806df60 2024-03-25T19:25:28.630 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.629+0000 7fa6caffd640 1 -- 172.21.15.83:0/2735269963 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7fa6cc054be0 con 0x7fa6d41430c0 2024-03-25T19:25:28.630 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.629+0000 7fa6caffd640 10 client.4479.objecter ms_dispatch 0x7fa6d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:25:28.630 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.629+0000 7fa6caffd640 3 client.4479.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:25:28.630 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.629+0000 7fa6caffd640 3 client.4479.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:25:28.630 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.629+0000 7fa6caffd640 20 client.4479.objecter dump_active .. 0 homeless 2024-03-25T19:25:28.630 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.629+0000 7fa6caffd640 1 -- 172.21.15.83:0/2735269963 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa6cc0506f0 con 0x7fa6d41430c0 2024-03-25T19:25:28.631 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.629+0000 7fa6caffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:28.631 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.629+0000 7fa6caffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:25:28.878 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.878+0000 7fa6dc7d0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:28.879 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.878+0000 7fa6dc7d0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:28.879 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.878+0000 7fa6dc7d0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:25:28.879 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.878+0000 7fa6dc7d0640 1 -- 172.21.15.83:0/2735269963 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa6d413df50 con 0x7fa6d41430c0 2024-03-25T19:25:28.921 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.921+0000 7fa6caffd640 1 -- 172.21.15.83:0/2735269963 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7fa6cc052620 con 0x7fa6d41430c0 2024-03-25T19:25:28.921 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.921+0000 7fa6caffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:28.921 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.921+0000 7fa6caffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:25:28.921 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:25:28.921 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:25:28.924 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.924+0000 7fa6dc7d0640 10 librados: watch_flush enter 2024-03-25T19:25:28.924 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.924+0000 7fa6dc7d0640 10 librados: watch_flush exit 2024-03-25T19:25:28.924 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.924+0000 7fa6dc7d0640 20 client.4479.objecter shutdown clearing up homeless session... 2024-03-25T19:25:28.924 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.924+0000 7fa6dc7d0640 10 client.4479.objecter successfully canceled tick 2024-03-25T19:25:28.924 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.924+0000 7fa6dc7d0640 1 -- 172.21.15.83:0/2735269963 >> v1:172.21.15.103:6800/345072327 conn(0x7fa6b806df60 legacy=0x7fa6b8070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:28.924 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.924+0000 7fa6dc7d0640 10 monclient: shutdown 2024-03-25T19:25:28.925 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.924+0000 7fa6dc7d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:28.925 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.924+0000 7fa6dc7d0640 1 -- 172.21.15.83:0/2735269963 >> v1:172.21.15.83:6789/0 conn(0x7fa6d41430c0 legacy=0x7fa6d41d7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:28.925 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.924+0000 7fa6dc7d0640 1 -- 172.21.15.83:0/2735269963 shutdown_connections 2024-03-25T19:25:28.925 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.924+0000 7fa6dc7d0640 1 -- 172.21.15.83:0/2735269963 >> 172.21.15.83:0/2735269963 conn(0x7fa6d40b7ea0 msgr2=0x7fa6d4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:28.925 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.924+0000 7fa6dc7d0640 1 -- 172.21.15.83:0/2735269963 shutdown_connections 2024-03-25T19:25:28.925 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.924+0000 7fa6dc7d0640 1 -- 172.21.15.83:0/2735269963 wait complete. 2024-03-25T19:25:28.925 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:28.924+0000 7fa6dc7d0640 1 librados: shutdown 2024-03-25T19:25:28.936 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:25:35.742 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:25:35.861 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.860+0000 7fcd1b625640 10 monclient: get_monmap_and_config 2024-03-25T19:25:35.861 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.860+0000 7fcd1b625640 10 monclient: build_initial_monmap 2024-03-25T19:25:35.861 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.860+0000 7fcd1b625640 10 monclient: monmap: 2024-03-25T19:25:35.861 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:25:35.861 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:25:35.861 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:25:35.862609+0000 2024-03-25T19:25:35.861 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:25:35.862609+0000 2024-03-25T19:25:35.862 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:35.862 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:25:35.862 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:25:35.862 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:25:35.862 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:25:35.862 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:35.862 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.861+0000 7fcd1b625640 1 Processor -- start 2024-03-25T19:25:35.863 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.861+0000 7fcd1b625640 1 -- start start 2024-03-25T19:25:35.863 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.861+0000 7fcd1b625640 10 monclient: init 2024-03-25T19:25:35.864 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.862+0000 7fcd1b625640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:35.864 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.862+0000 7fcd1b625640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:25:35.864 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.862+0000 7fcd1b625640 10 monclient(hunting): picked mon.noname-b con 0x7fcd140a98f0 addr v1:172.21.15.83:6790/0 2024-03-25T19:25:35.864 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.862+0000 7fcd1b625640 10 monclient(hunting): picked mon.noname-a con 0x7fcd1413f710 addr v1:172.21.15.83:6789/0 2024-03-25T19:25:35.864 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.862+0000 7fcd1b625640 10 monclient(hunting): picked mon.noname-c con 0x7fcd141430c0 addr v1:172.21.15.103:6789/0 2024-03-25T19:25:35.864 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.862+0000 7fcd1b625640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd14142520 con 0x7fcd1413f710 2024-03-25T19:25:35.864 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.862+0000 7fcd1b625640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd140b28a0 con 0x7fcd141430c0 2024-03-25T19:25:35.864 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.862+0000 7fcd1b625640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd140b2a30 con 0x7fcd140a98f0 2024-03-25T19:25:35.864 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.862+0000 7fcd1b625640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:35.864 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.862+0000 7fcd1b625640 10 monclient(hunting): authenticate will time out at 6393.135710s 2024-03-25T19:25:35.869 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.868+0000 7fcd18bc0640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fcd1413f710 0x7fcd141429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:53336/0 (socket says 172.21.15.83:53336) 2024-03-25T19:25:35.869 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.868+0000 7fcd18bc0640 1 -- 172.21.15.83:0/1255744236 learned_addr learned my addr 172.21.15.83:0/1255744236 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:25:35.886 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.885+0000 7fcd13fff640 1 -- 172.21.15.83:0/1255744236 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2858758950 0 0) 0x7fcd14142520 con 0x7fcd1413f710 2024-03-25T19:25:35.886 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.885+0000 7fcd13fff640 10 monclient(hunting): _negotiate 2024-03-25T19:25:35.886 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.885+0000 7fcd13fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:35.886 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.885+0000 7fcd13fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:35.886 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.885+0000 7fcd13fff640 10 monclient(hunting): my global_id is 4482 2024-03-25T19:25:35.886 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.885+0000 7fcd13fff640 1 -- 172.21.15.83:0/1255744236 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccfc003520 con 0x7fcd1413f710 2024-03-25T19:25:35.887 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.886+0000 7fcd13fff640 1 -- 172.21.15.83:0/1255744236 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1746318611 0 0) 0x7fcd140b28a0 con 0x7fcd141430c0 2024-03-25T19:25:35.887 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.886+0000 7fcd13fff640 10 monclient(hunting): _negotiate 2024-03-25T19:25:35.887 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.886+0000 7fcd13fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:35.887 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.886+0000 7fcd13fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:35.887 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.886+0000 7fcd13fff640 10 monclient(hunting): my global_id is 4492 2024-03-25T19:25:35.887 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.886+0000 7fcd13fff640 1 -- 172.21.15.83:0/1255744236 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd14142520 con 0x7fcd141430c0 2024-03-25T19:25:35.892 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.891+0000 7fcd13fff640 1 -- 172.21.15.83:0/1255744236 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2635089271 0 0) 0x7fcd140b2a30 con 0x7fcd140a98f0 2024-03-25T19:25:35.892 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.891+0000 7fcd13fff640 10 monclient(hunting): _negotiate 2024-03-25T19:25:35.892 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.891+0000 7fcd13fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:35.892 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.891+0000 7fcd13fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:35.892 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.891+0000 7fcd13fff640 10 monclient(hunting): my global_id is 4499 2024-03-25T19:25:35.892 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.891+0000 7fcd13fff640 1 -- 172.21.15.83:0/1255744236 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd140b28a0 con 0x7fcd140a98f0 2024-03-25T19:25:35.899 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.897+0000 7fcd13fff640 1 -- 172.21.15.83:0/1255744236 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 886957670 0 0) 0x7fcd14142520 con 0x7fcd141430c0 2024-03-25T19:25:35.899 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.897+0000 7fcd13fff640 1 -- 172.21.15.83:0/1255744236 >> v1:172.21.15.83:6790/0 conn(0x7fcd140a98f0 legacy=0x7fcd140a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:35.899 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.897+0000 7fcd13fff640 1 -- 172.21.15.83:0/1255744236 >> v1:172.21.15.83:6789/0 conn(0x7fcd1413f710 legacy=0x7fcd141429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:35.899 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.897+0000 7fcd13fff640 10 monclient: _finish_hunting 0 2024-03-25T19:25:35.899 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.897+0000 7fcd13fff640 1 monclient: found mon.noname-c 2024-03-25T19:25:35.899 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.897+0000 7fcd13fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:35.899 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.897+0000 7fcd13fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.103:6789/0 2024-03-25T19:25:35.899 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.897+0000 7fcd13fff640 1 -- 172.21.15.83:0/1255744236 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd140b44c0 con 0x7fcd141430c0 2024-03-25T19:25:35.899 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.897+0000 7fcd13fff640 10 monclient: _finish_auth 0 2024-03-25T19:25:35.899 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.897+0000 7fcd13fff640 10 monclient: _check_auth_tickets 2024-03-25T19:25:35.899 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.897+0000 7fcd13fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:35.900 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.897+0000 7fcd13fff640 1 -- 172.21.15.83:0/1255744236 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fcd0c0019f0 con 0x7fcd141430c0 2024-03-25T19:25:35.900 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.897+0000 7fcd13fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:35.900 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.897+0000 7fcd13fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:25:35.900 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.897+0000 7fcd13fff640 10 monclient: dump: 2024-03-25T19:25:35.900 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:25:35.900 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:25:35.900 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:35.900 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:35.900 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:35.900 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:25:35.900 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:25:35.900 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:25:35.900 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:25:35.901 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:35.901 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.897+0000 7fcd1b625640 5 monclient: authenticate success, global_id 4492 2024-03-25T19:25:35.901 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.897+0000 7fcd1b625640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:25:35.909 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.909+0000 7fcd13fff640 1 -- 172.21.15.83:0/1255744236 <== mon.1 v1:172.21.15.103:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcd0c001b50 con 0x7fcd141430c0 2024-03-25T19:25:35.909 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.909+0000 7fcd13fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:35.909 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.909+0000 7fcd13fff640 1 -- 172.21.15.83:0/1255744236 <== mon.1 v1:172.21.15.103:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fcd0c001f60 con 0x7fcd141430c0 2024-03-25T19:25:35.909 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.909+0000 7fcd13fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:35.910 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.909+0000 7fcd13fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:25:35.910 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.909+0000 7fcd13fff640 10 monclient: dump: 2024-03-25T19:25:35.910 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:25:35.910 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:25:35.910 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:35.910 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:35.910 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:35.910 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:25:35.910 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:25:35.910 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:25:35.910 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:25:35.910 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:35.910 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.909+0000 7fcd1b625640 10 monclient: get_monmap_and_config success 2024-03-25T19:25:35.910 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.909+0000 7fcd1b625640 4 set_mon_vals no callback set 2024-03-25T19:25:35.912 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.910+0000 7fcd1b625640 10 monclient: shutdown 2024-03-25T19:25:35.912 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.910+0000 7fcd1b625640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:35.912 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.910+0000 7fcd1b625640 1 -- 172.21.15.83:0/1255744236 >> v1:172.21.15.103:6789/0 conn(0x7fcd141430c0 legacy=0x7fcd1414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:35.912 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.910+0000 7fcd1b625640 1 -- 172.21.15.83:0/1255744236 shutdown_connections 2024-03-25T19:25:35.912 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.910+0000 7fcd1b625640 1 -- 172.21.15.83:0/1255744236 >> 172.21.15.83:0/1255744236 conn(0x7fcd140b7ea0 msgr2=0x7fcd140b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:35.912 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.910+0000 7fcd1b625640 1 -- 172.21.15.83:0/1255744236 shutdown_connections 2024-03-25T19:25:35.912 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.910+0000 7fcd1b625640 1 -- 172.21.15.83:0/1255744236 wait complete. 2024-03-25T19:25:35.912 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.911+0000 7fcd1b625640 10 monclient: build_initial_monmap 2024-03-25T19:25:35.912 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.911+0000 7fcd1b625640 10 monclient: monmap: 2024-03-25T19:25:35.912 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:25:35.912 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:25:35.912 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:25:35.912 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:25:35.913 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:35.913 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:25:35.913 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:25:35.913 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:25:35.913 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:25:35.913 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:35.913 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.911+0000 7fcd1b625640 1 librados: starting msgr at 2024-03-25T19:25:35.913 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.911+0000 7fcd1b625640 1 librados: starting objecter 2024-03-25T19:25:35.913 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.911+0000 7fcd1b625640 1 Processor -- start 2024-03-25T19:25:35.913 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.911+0000 7fcd1b625640 1 -- start start 2024-03-25T19:25:35.913 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.911+0000 7fcd1b625640 1 librados: setting wanted keys 2024-03-25T19:25:35.913 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.911+0000 7fcd1b625640 1 librados: calling monclient init 2024-03-25T19:25:35.913 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.911+0000 7fcd1b625640 10 monclient: init 2024-03-25T19:25:35.914 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.912+0000 7fcd1b625640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:35.914 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.912+0000 7fcd1b625640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:25:35.914 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.912+0000 7fcd1b625640 10 monclient(hunting): picked mon.noname-c con 0x7fcd140a98f0 addr v1:172.21.15.83:6790/0 2024-03-25T19:25:35.914 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.912+0000 7fcd1b625640 10 monclient(hunting): picked mon.noname-b con 0x7fcd1413f710 addr v1:172.21.15.103:6789/0 2024-03-25T19:25:35.915 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.912+0000 7fcd1b625640 10 monclient(hunting): picked mon.noname-a con 0x7fcd141430c0 addr v1:172.21.15.83:6789/0 2024-03-25T19:25:35.915 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.912+0000 7fcd1b625640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd141388a0 con 0x7fcd141430c0 2024-03-25T19:25:35.915 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.912+0000 7fcd1b625640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd141349e0 con 0x7fcd1413f710 2024-03-25T19:25:35.915 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.912+0000 7fcd1b625640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd14138d30 con 0x7fcd140a98f0 2024-03-25T19:25:35.915 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.912+0000 7fcd1b625640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:35.915 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.912+0000 7fcd1b625640 10 monclient(hunting): authenticate will time out at 6393.185141s 2024-03-25T19:25:35.917 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.916+0000 7fcd18bc0640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fcd1413f710 0x7fcd14133bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:55438/0 (socket says 172.21.15.83:55438) 2024-03-25T19:25:35.917 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.916+0000 7fcd18bc0640 1 -- 172.21.15.83:0/2656196638 learned_addr learned my addr 172.21.15.83:0/2656196638 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:25:35.917 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.917+0000 7fcd11ffb640 10 client.?.objecter ms_handle_connect 0x7fcd1413f710 2024-03-25T19:25:35.917 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.917+0000 7fcd11ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:35.926 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.925+0000 7fcd11ffb640 1 -- 172.21.15.83:0/2656196638 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2702796770 0 0) 0x7fcd141349e0 con 0x7fcd1413f710 2024-03-25T19:25:35.926 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.925+0000 7fcd11ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:25:35.926 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.925+0000 7fcd11ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:35.926 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.925+0000 7fcd11ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:35.926 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.925+0000 7fcd11ffb640 10 monclient(hunting): my global_id is 4495 2024-03-25T19:25:35.926 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.925+0000 7fcd11ffb640 1 -- 172.21.15.83:0/2656196638 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd00004eb0 con 0x7fcd1413f710 2024-03-25T19:25:35.926 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.925+0000 7fcd11ffb640 10 client.?.objecter ms_handle_connect 0x7fcd141430c0 2024-03-25T19:25:35.926 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.925+0000 7fcd11ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:35.932 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.931+0000 7fcd11ffb640 1 -- 172.21.15.83:0/2656196638 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1817012293 0 0) 0x7fcd141388a0 con 0x7fcd141430c0 2024-03-25T19:25:35.932 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.931+0000 7fcd11ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:25:35.932 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.931+0000 7fcd11ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:35.932 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.931+0000 7fcd11ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:35.932 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.931+0000 7fcd11ffb640 10 monclient(hunting): my global_id is 4485 2024-03-25T19:25:35.932 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.931+0000 7fcd11ffb640 1 -- 172.21.15.83:0/2656196638 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd141349e0 con 0x7fcd141430c0 2024-03-25T19:25:35.933 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.933+0000 7fcd11ffb640 10 client.?.objecter ms_handle_connect 0x7fcd140a98f0 2024-03-25T19:25:35.933 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.933+0000 7fcd11ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:35.935 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.934+0000 7fcd11ffb640 1 -- 172.21.15.83:0/2656196638 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1269293904 0 0) 0x7fcd00004eb0 con 0x7fcd1413f710 2024-03-25T19:25:35.937 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.934+0000 7fcd11ffb640 1 -- 172.21.15.83:0/2656196638 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcd141388a0 con 0x7fcd1413f710 2024-03-25T19:25:35.937 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.934+0000 7fcd11ffb640 1 -- 172.21.15.83:0/2656196638 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fcd08001950 con 0x7fcd1413f710 2024-03-25T19:25:35.937 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.934+0000 7fcd11ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:35.937 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.934+0000 7fcd11ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:25:35.937 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.934+0000 7fcd11ffb640 10 monclient(hunting): dump: 2024-03-25T19:25:35.937 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:25:35.937 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:25:35.938 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:35.938 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:35.938 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:35.938 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:25:35.938 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:25:35.938 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:25:35.938 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:25:35.938 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:35.938 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.934+0000 7fcd11ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:25:35.940 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.939+0000 7fcd11ffb640 1 -- 172.21.15.83:0/2656196638 <== mon.1 v1:172.21.15.103:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3780855520 0 0) 0x7fcd141388a0 con 0x7fcd1413f710 2024-03-25T19:25:35.940 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.939+0000 7fcd11ffb640 1 -- 172.21.15.83:0/2656196638 >> v1:172.21.15.83:6790/0 conn(0x7fcd140a98f0 legacy=0x7fcd141da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:35.940 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.939+0000 7fcd11ffb640 1 -- 172.21.15.83:0/2656196638 >> v1:172.21.15.83:6789/0 conn(0x7fcd141430c0 legacy=0x7fcd141342d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:35.940 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.939+0000 7fcd11ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:25:35.940 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.939+0000 7fcd11ffb640 1 monclient: found mon.b 2024-03-25T19:25:35.940 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.939+0000 7fcd11ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:35.940 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.939+0000 7fcd11ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:25:35.940 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.939+0000 7fcd11ffb640 1 -- 172.21.15.83:0/2656196638 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd140aba50 con 0x7fcd1413f710 2024-03-25T19:25:35.940 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.939+0000 7fcd11ffb640 10 monclient: _finish_auth 0 2024-03-25T19:25:35.940 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.939+0000 7fcd11ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:25:35.941 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.939+0000 7fcd11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:35.941 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.939+0000 7fcd1b625640 5 monclient: authenticate success, global_id 4495 2024-03-25T19:25:35.941 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.939+0000 7fcd1b625640 10 monclient: _renew_subs 2024-03-25T19:25:35.941 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.939+0000 7fcd1b625640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:25:35.941 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.939+0000 7fcd1b625640 1 -- 172.21.15.83:0/2656196638 --> v1:172.21.15.103:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd140b1770 con 0x7fcd1413f710 2024-03-25T19:25:35.941 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.939+0000 7fcd1b625640 10 client.4495.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:25:35.941 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.939+0000 7fcd1b625640 10 monclient: _renew_subs 2024-03-25T19:25:35.941 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.939+0000 7fcd1b625640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:25:35.941 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.939+0000 7fcd1b625640 1 -- 172.21.15.83:0/2656196638 --> v1:172.21.15.103:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd14137bf0 con 0x7fcd1413f710 2024-03-25T19:25:35.941 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.939+0000 7fcd1b625640 1 librados: init done 2024-03-25T19:25:35.942 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.940+0000 7fcd1b625640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:35.942 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.940+0000 7fcd1b625640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:35.942 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.940+0000 7fcd1b625640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:25:35.942 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:35.940+0000 7fcd1b625640 1 -- 172.21.15.83:0/2656196638 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd140627b0 con 0x7fcd1413f710 2024-03-25T19:25:36.231 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:36.230+0000 7fcd11ffb640 1 -- 172.21.15.83:0/2656196638 <== mon.1 v1:172.21.15.103:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcd080047e0 con 0x7fcd1413f710 2024-03-25T19:25:36.231 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:36.230+0000 7fcd11ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:36.231 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:36.230+0000 7fcd11ffb640 1 -- 172.21.15.83:0/2656196638 <== mon.1 v1:172.21.15.103:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fcd08001d80 con 0x7fcd1413f710 2024-03-25T19:25:36.231 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:36.230+0000 7fcd11ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:36.231 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:36.230+0000 7fcd11ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:25:36.231 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:36.230+0000 7fcd11ffb640 10 monclient: dump: 2024-03-25T19:25:36.231 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:25:36.231 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:25:36.231 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:36.231 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:36.232 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:36.232 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:25:36.232 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:25:36.232 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:25:36.232 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:25:36.232 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:36.232 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:36.230+0000 7fcd137fe640 4 set_mon_vals no callback set 2024-03-25T19:25:36.232 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:36.230+0000 7fcd11ffb640 1 -- 172.21.15.83:0/2656196638 <== mon.1 v1:172.21.15.103:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7fcd08004c30 con 0x7fcd1413f710 2024-03-25T19:25:36.233 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:36.231+0000 7fcd193c1640 10 monclient: get_auth_request con 0x7fcd0006dd10 auth_method 0 2024-03-25T19:25:36.233 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:36.231+0000 7fcd11ffb640 10 client.4495.objecter ms_handle_connect 0x7fcd0006dd10 2024-03-25T19:25:36.236 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:36.236+0000 7fcd11ffb640 1 -- 172.21.15.83:0/2656196638 <== mon.1 v1:172.21.15.103:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7fcd08054820 con 0x7fcd1413f710 2024-03-25T19:25:36.237 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:36.236+0000 7fcd11ffb640 10 client.4495.objecter ms_dispatch 0x7fcd14000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:25:36.237 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:36.236+0000 7fcd11ffb640 3 client.4495.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:25:36.237 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:36.236+0000 7fcd11ffb640 3 client.4495.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:25:36.237 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:36.236+0000 7fcd11ffb640 20 client.4495.objecter dump_active .. 0 homeless 2024-03-25T19:25:36.237 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:36.236+0000 7fcd11ffb640 1 -- 172.21.15.83:0/2656196638 <== mon.1 v1:172.21.15.103:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcd08050160 con 0x7fcd1413f710 2024-03-25T19:25:36.237 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:36.236+0000 7fcd11ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:36.237 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:36.236+0000 7fcd11ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:25:36.456 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:36.455+0000 7fcd1b625640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:36.456 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:36.455+0000 7fcd1b625640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:36.456 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:36.455+0000 7fcd1b625640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:25:36.456 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:36.455+0000 7fcd1b625640 1 -- 172.21.15.83:0/2656196638 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcd14138a50 con 0x7fcd1413f710 2024-03-25T19:25:36.480 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:36.479+0000 7fcd11ffb640 1 -- 172.21.15.83:0/2656196638 <== mon.1 v1:172.21.15.103:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7fcd14138a50 con 0x7fcd1413f710 2024-03-25T19:25:36.480 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:36.479+0000 7fcd11ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:36.480 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:36.479+0000 7fcd11ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:25:36.480 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:25:36.480 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:25:36.483 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:36.482+0000 7fcd1b625640 10 librados: watch_flush enter 2024-03-25T19:25:36.483 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:36.482+0000 7fcd1b625640 10 librados: watch_flush exit 2024-03-25T19:25:36.483 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:36.482+0000 7fcd1b625640 20 client.4495.objecter shutdown clearing up homeless session... 2024-03-25T19:25:36.483 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:36.482+0000 7fcd1b625640 10 client.4495.objecter successfully canceled tick 2024-03-25T19:25:36.483 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:36.482+0000 7fcd1b625640 1 -- 172.21.15.83:0/2656196638 >> v1:172.21.15.103:6800/345072327 conn(0x7fcd0006dd10 legacy=0x7fcd000701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:36.483 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:36.482+0000 7fcd1b625640 10 monclient: shutdown 2024-03-25T19:25:36.483 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:36.482+0000 7fcd1b625640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:36.484 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:36.482+0000 7fcd1b625640 1 -- 172.21.15.83:0/2656196638 >> v1:172.21.15.103:6789/0 conn(0x7fcd1413f710 legacy=0x7fcd14133bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:36.484 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:36.482+0000 7fcd1b625640 1 -- 172.21.15.83:0/2656196638 shutdown_connections 2024-03-25T19:25:36.484 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:36.482+0000 7fcd1b625640 1 -- 172.21.15.83:0/2656196638 >> 172.21.15.83:0/2656196638 conn(0x7fcd140b7ea0 msgr2=0x7fcd14074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:36.484 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:36.482+0000 7fcd1b625640 1 -- 172.21.15.83:0/2656196638 shutdown_connections 2024-03-25T19:25:36.484 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:36.482+0000 7fcd1b625640 1 -- 172.21.15.83:0/2656196638 wait complete. 2024-03-25T19:25:36.484 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:36.482+0000 7fcd1b625640 1 librados: shutdown 2024-03-25T19:25:36.494 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:25:43.300 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:25:43.410 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.411+0000 7fc31178f640 10 monclient: get_monmap_and_config 2024-03-25T19:25:43.411 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.411+0000 7fc31178f640 10 monclient: build_initial_monmap 2024-03-25T19:25:43.411 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.411+0000 7fc31178f640 10 monclient: monmap: 2024-03-25T19:25:43.411 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:25:43.411 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:25:43.411 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:25:43.412198+0000 2024-03-25T19:25:43.411 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:25:43.412198+0000 2024-03-25T19:25:43.411 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:43.411 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:25:43.411 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:25:43.411 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:25:43.411 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:25:43.412 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:43.413 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.411+0000 7fc31178f640 1 Processor -- start 2024-03-25T19:25:43.413 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.412+0000 7fc31178f640 1 -- start start 2024-03-25T19:25:43.413 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.412+0000 7fc31178f640 10 monclient: init 2024-03-25T19:25:43.413 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.412+0000 7fc31178f640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:43.413 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.412+0000 7fc31178f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:25:43.413 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.412+0000 7fc31178f640 10 monclient(hunting): picked mon.noname-a con 0x7fc30c0b4d60 addr v1:172.21.15.83:6789/0 2024-03-25T19:25:43.413 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.412+0000 7fc31178f640 10 monclient(hunting): picked mon.noname-c con 0x7fc30c0b1730 addr v1:172.21.15.103:6789/0 2024-03-25T19:25:43.413 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.412+0000 7fc31178f640 10 monclient(hunting): picked mon.noname-b con 0x7fc30c14be90 addr v1:172.21.15.83:6790/0 2024-03-25T19:25:43.413 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.412+0000 7fc31178f640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc30c0af940 con 0x7fc30c0b4d60 2024-03-25T19:25:43.413 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.412+0000 7fc31178f640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc30c0b0a70 con 0x7fc30c0b1730 2024-03-25T19:25:43.413 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.412+0000 7fc31178f640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc30c0b0d50 con 0x7fc30c14be90 2024-03-25T19:25:43.413 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.412+0000 7fc31178f640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:43.414 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.412+0000 7fc31178f640 10 monclient(hunting): authenticate will time out at 6400.684648s 2024-03-25T19:25:43.420 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.420+0000 7fc30a7fc640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fc30c0b1730 0x7fc30c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:60124/0 (socket says 172.21.15.83:60124) 2024-03-25T19:25:43.420 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.420+0000 7fc30a7fc640 1 -- 172.21.15.83:0/1196853680 learned_addr learned my addr 172.21.15.83:0/1196853680 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:25:43.430 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.430+0000 7fc309ffb640 1 -- 172.21.15.83:0/1196853680 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 70797218 0 0) 0x7fc30c0af940 con 0x7fc30c0b4d60 2024-03-25T19:25:43.430 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.430+0000 7fc309ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:25:43.430 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.430+0000 7fc309ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:43.430 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.430+0000 7fc309ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:43.430 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.430+0000 7fc309ffb640 10 monclient(hunting): my global_id is 4488 2024-03-25T19:25:43.430 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.430+0000 7fc309ffb640 1 -- 172.21.15.83:0/1196853680 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2f4003520 con 0x7fc30c0b4d60 2024-03-25T19:25:43.436 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.436+0000 7fc309ffb640 1 -- 172.21.15.83:0/1196853680 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2098346441 0 0) 0x7fc2f4003520 con 0x7fc30c0b4d60 2024-03-25T19:25:43.436 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.436+0000 7fc309ffb640 1 -- 172.21.15.83:0/1196853680 >> v1:172.21.15.83:6790/0 conn(0x7fc30c14be90 legacy=0x7fc30c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:43.436 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.436+0000 7fc309ffb640 1 -- 172.21.15.83:0/1196853680 >> v1:172.21.15.103:6789/0 conn(0x7fc30c0b1730 legacy=0x7fc30c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:43.436 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.436+0000 7fc309ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:25:43.437 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.436+0000 7fc309ffb640 1 monclient: found mon.noname-a 2024-03-25T19:25:43.437 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.436+0000 7fc309ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:43.437 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.436+0000 7fc309ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.83:6789/0 2024-03-25T19:25:43.437 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.436+0000 7fc309ffb640 1 -- 172.21.15.83:0/1196853680 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc30c0b1030 con 0x7fc30c0b4d60 2024-03-25T19:25:43.437 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.436+0000 7fc309ffb640 10 monclient: _finish_auth 0 2024-03-25T19:25:43.437 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.436+0000 7fc309ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:25:43.437 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.436+0000 7fc309ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:43.437 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.436+0000 7fc309ffb640 1 -- 172.21.15.83:0/1196853680 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fc2fc001690 con 0x7fc30c0b4d60 2024-03-25T19:25:43.437 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.436+0000 7fc31178f640 5 monclient: authenticate success, global_id 4488 2024-03-25T19:25:43.437 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.436+0000 7fc31178f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:25:43.437 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.436+0000 7fc309ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:43.437 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.436+0000 7fc309ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:25:43.437 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.436+0000 7fc309ffb640 10 monclient: dump: 2024-03-25T19:25:43.437 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:25:43.437 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:25:43.437 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:43.438 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:43.438 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:43.438 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:25:43.438 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:25:43.438 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:25:43.438 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:25:43.438 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:43.438 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.436+0000 7fc31178f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:25:43.465 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.466+0000 7fc309ffb640 1 -- 172.21.15.83:0/1196853680 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc2fc0017f0 con 0x7fc30c0b4d60 2024-03-25T19:25:43.465 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.466+0000 7fc309ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:43.466 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.466+0000 7fc309ffb640 1 -- 172.21.15.83:0/1196853680 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fc2fc001c00 con 0x7fc30c0b4d60 2024-03-25T19:25:43.466 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.466+0000 7fc31178f640 10 monclient: get_monmap_and_config success 2024-03-25T19:25:43.466 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.466+0000 7fc31178f640 4 set_mon_vals no callback set 2024-03-25T19:25:43.466 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.466+0000 7fc309ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:43.466 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.466+0000 7fc309ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:25:43.466 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.466+0000 7fc309ffb640 10 monclient: dump: 2024-03-25T19:25:43.466 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:25:43.466 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:25:43.466 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:43.466 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:43.466 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:43.466 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:25:43.466 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:25:43.467 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:25:43.467 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:25:43.467 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:43.469 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.467+0000 7fc31178f640 10 monclient: shutdown 2024-03-25T19:25:43.469 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.467+0000 7fc31178f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:43.469 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.467+0000 7fc31178f640 1 -- 172.21.15.83:0/1196853680 >> v1:172.21.15.83:6789/0 conn(0x7fc30c0b4d60 legacy=0x7fc30c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:43.469 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.467+0000 7fc31178f640 1 -- 172.21.15.83:0/1196853680 shutdown_connections 2024-03-25T19:25:43.469 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.467+0000 7fc31178f640 1 -- 172.21.15.83:0/1196853680 >> 172.21.15.83:0/1196853680 conn(0x7fc30c0617a0 msgr2=0x7fc30c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:43.469 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.467+0000 7fc31178f640 1 -- 172.21.15.83:0/1196853680 shutdown_connections 2024-03-25T19:25:43.469 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.467+0000 7fc31178f640 1 -- 172.21.15.83:0/1196853680 wait complete. 2024-03-25T19:25:43.469 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.467+0000 7fc31178f640 10 monclient: build_initial_monmap 2024-03-25T19:25:43.469 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.467+0000 7fc31178f640 10 monclient: monmap: 2024-03-25T19:25:43.469 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:25:43.470 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:25:43.470 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:25:43.470 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:25:43.470 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:43.470 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:25:43.470 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:25:43.470 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:25:43.470 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:25:43.470 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:43.470 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.468+0000 7fc31178f640 1 librados: starting msgr at 2024-03-25T19:25:43.470 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.468+0000 7fc31178f640 1 librados: starting objecter 2024-03-25T19:25:43.470 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.468+0000 7fc31178f640 1 Processor -- start 2024-03-25T19:25:43.470 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.468+0000 7fc31178f640 1 -- start start 2024-03-25T19:25:43.470 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.468+0000 7fc31178f640 1 librados: setting wanted keys 2024-03-25T19:25:43.471 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.468+0000 7fc31178f640 1 librados: calling monclient init 2024-03-25T19:25:43.471 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.468+0000 7fc31178f640 10 monclient: init 2024-03-25T19:25:43.471 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.469+0000 7fc31178f640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:43.471 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.469+0000 7fc31178f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:25:43.471 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.469+0000 7fc31178f640 10 monclient(hunting): picked mon.noname-c con 0x7fc30c0b1730 addr v1:172.21.15.83:6790/0 2024-03-25T19:25:43.471 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.469+0000 7fc31178f640 10 monclient(hunting): picked mon.noname-a con 0x7fc30c0b4d60 addr v1:172.21.15.83:6789/0 2024-03-25T19:25:43.471 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.469+0000 7fc31178f640 10 monclient(hunting): picked mon.noname-b con 0x7fc30c14be90 addr v1:172.21.15.103:6789/0 2024-03-25T19:25:43.471 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.469+0000 7fc31178f640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc30c142760 con 0x7fc30c0b4d60 2024-03-25T19:25:43.471 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.469+0000 7fc31178f640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc30c139cf0 con 0x7fc30c14be90 2024-03-25T19:25:43.471 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.469+0000 7fc31178f640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc30c139fd0 con 0x7fc30c0b1730 2024-03-25T19:25:43.471 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.469+0000 7fc31178f640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:43.471 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.469+0000 7fc31178f640 10 monclient(hunting): authenticate will time out at 6400.741185s 2024-03-25T19:25:43.480 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.480+0000 7fc30affd640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fc30c0b1730 0x7fc30c140380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:37862/0 (socket says 172.21.15.83:37862) 2024-03-25T19:25:43.480 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.480+0000 7fc30affd640 1 -- 172.21.15.83:0/1971898268 learned_addr learned my addr 172.21.15.83:0/1971898268 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:25:43.480 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.481+0000 7fc2f37fe640 10 client.?.objecter ms_handle_connect 0x7fc30c0b1730 2024-03-25T19:25:43.480 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.481+0000 7fc2f37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:43.482 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.483+0000 7fc2f37fe640 10 client.?.objecter ms_handle_connect 0x7fc30c14be90 2024-03-25T19:25:43.482 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.483+0000 7fc2f37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:43.495 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.495+0000 7fc2f37fe640 1 -- 172.21.15.83:0/1971898268 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3080607772 0 0) 0x7fc30c139cf0 con 0x7fc30c14be90 2024-03-25T19:25:43.495 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.495+0000 7fc2f37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:25:43.495 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.495+0000 7fc2f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:43.495 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.495+0000 7fc2f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:43.495 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.495+0000 7fc2f37fe640 10 monclient(hunting): my global_id is 4501 2024-03-25T19:25:43.495 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.495+0000 7fc2f37fe640 1 -- 172.21.15.83:0/1971898268 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2f8004eb0 con 0x7fc30c14be90 2024-03-25T19:25:43.498 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.498+0000 7fc2f37fe640 1 -- 172.21.15.83:0/1971898268 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1489667339 0 0) 0x7fc30c139fd0 con 0x7fc30c0b1730 2024-03-25T19:25:43.498 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.498+0000 7fc2f37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:25:43.498 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.498+0000 7fc2f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:43.498 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.498+0000 7fc2f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:43.498 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.498+0000 7fc2f37fe640 10 monclient(hunting): my global_id is 4508 2024-03-25T19:25:43.498 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.498+0000 7fc2f37fe640 1 -- 172.21.15.83:0/1971898268 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc30c139cf0 con 0x7fc30c0b1730 2024-03-25T19:25:43.503 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.504+0000 7fc2f37fe640 10 client.?.objecter ms_handle_connect 0x7fc30c0b4d60 2024-03-25T19:25:43.503 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.504+0000 7fc2f37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:43.512 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.512+0000 7fc2f37fe640 1 -- 172.21.15.83:0/1971898268 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3293296674 0 0) 0x7fc2f8004eb0 con 0x7fc30c14be90 2024-03-25T19:25:43.512 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.513+0000 7fc2f37fe640 1 -- 172.21.15.83:0/1971898268 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc30c139fd0 con 0x7fc30c14be90 2024-03-25T19:25:43.512 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.513+0000 7fc2f37fe640 1 -- 172.21.15.83:0/1971898268 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fc304001b90 con 0x7fc30c14be90 2024-03-25T19:25:43.512 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.513+0000 7fc2f37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:43.513 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.513+0000 7fc2f37fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:25:43.513 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.513+0000 7fc2f37fe640 10 monclient(hunting): dump: 2024-03-25T19:25:43.513 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:25:43.513 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:25:43.513 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:43.513 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:43.513 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:43.513 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:25:43.513 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:25:43.513 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:25:43.513 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:25:43.513 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:43.513 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.513+0000 7fc2f37fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:25:43.516 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.517+0000 7fc2f37fe640 1 -- 172.21.15.83:0/1971898268 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2041052386 0 0) 0x7fc30c142760 con 0x7fc30c0b4d60 2024-03-25T19:25:43.517 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.517+0000 7fc2f37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:25:43.517 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.517+0000 7fc2f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:43.517 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.517+0000 7fc2f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:43.517 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.517+0000 7fc2f37fe640 10 monclient(hunting): my global_id is 4491 2024-03-25T19:25:43.517 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.517+0000 7fc2f37fe640 1 -- 172.21.15.83:0/1971898268 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2f8004eb0 con 0x7fc30c0b4d60 2024-03-25T19:25:43.527 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.527+0000 7fc2f37fe640 1 -- 172.21.15.83:0/1971898268 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 487346553 0 0) 0x7fc30c139cf0 con 0x7fc30c0b1730 2024-03-25T19:25:43.527 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.527+0000 7fc2f37fe640 1 -- 172.21.15.83:0/1971898268 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc30c142760 con 0x7fc30c0b1730 2024-03-25T19:25:43.527 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.527+0000 7fc2f37fe640 1 -- 172.21.15.83:0/1971898268 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fc2fc001c00 con 0x7fc30c0b1730 2024-03-25T19:25:43.527 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.527+0000 7fc2f37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:43.527 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.527+0000 7fc2f37fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:25:43.527 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.527+0000 7fc2f37fe640 10 monclient(hunting): dump: 2024-03-25T19:25:43.527 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:25:43.527 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:25:43.528 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:43.528 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:43.528 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:43.528 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:25:43.528 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:25:43.528 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:25:43.528 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:25:43.528 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:43.532 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.531+0000 7fc2f37fe640 1 -- 172.21.15.83:0/1971898268 <== mon.1 v1:172.21.15.103:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2728608101 0 0) 0x7fc30c139fd0 con 0x7fc30c14be90 2024-03-25T19:25:43.532 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.531+0000 7fc2f37fe640 1 -- 172.21.15.83:0/1971898268 >> v1:172.21.15.83:6790/0 conn(0x7fc30c0b1730 legacy=0x7fc30c140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:43.532 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.531+0000 7fc2f37fe640 1 -- 172.21.15.83:0/1971898268 >> v1:172.21.15.83:6789/0 conn(0x7fc30c0b4d60 legacy=0x7fc30c134530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:43.532 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.531+0000 7fc2f37fe640 10 monclient: _finish_hunting 0 2024-03-25T19:25:43.532 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.531+0000 7fc2f37fe640 1 monclient: found mon.b 2024-03-25T19:25:43.532 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.531+0000 7fc2f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:43.532 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.531+0000 7fc2f37fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:25:43.532 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.531+0000 7fc2f37fe640 1 -- 172.21.15.83:0/1971898268 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc30c139000 con 0x7fc30c14be90 2024-03-25T19:25:43.533 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.531+0000 7fc2f37fe640 10 monclient: _finish_auth 0 2024-03-25T19:25:43.533 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.531+0000 7fc2f37fe640 10 monclient: _check_auth_tickets 2024-03-25T19:25:43.533 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.531+0000 7fc2f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:43.533 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.531+0000 7fc31178f640 5 monclient: authenticate success, global_id 4501 2024-03-25T19:25:43.533 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.531+0000 7fc31178f640 10 monclient: _renew_subs 2024-03-25T19:25:43.533 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.531+0000 7fc31178f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:25:43.533 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.531+0000 7fc31178f640 1 -- 172.21.15.83:0/1971898268 --> v1:172.21.15.103:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc30c1430d0 con 0x7fc30c14be90 2024-03-25T19:25:43.533 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.531+0000 7fc31178f640 10 client.4501.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:25:43.533 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.531+0000 7fc31178f640 10 monclient: _renew_subs 2024-03-25T19:25:43.533 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.531+0000 7fc31178f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:25:43.533 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.531+0000 7fc31178f640 1 -- 172.21.15.83:0/1971898268 --> v1:172.21.15.103:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc30c140c70 con 0x7fc30c14be90 2024-03-25T19:25:43.533 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.531+0000 7fc31178f640 1 librados: init done 2024-03-25T19:25:43.533 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.532+0000 7fc31178f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:43.533 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.532+0000 7fc31178f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:43.534 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.532+0000 7fc31178f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:25:43.534 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.532+0000 7fc31178f640 1 -- 172.21.15.83:0/1971898268 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc30c061b80 con 0x7fc30c14be90 2024-03-25T19:25:43.816 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.816+0000 7fc2f37fe640 1 -- 172.21.15.83:0/1971898268 <== mon.1 v1:172.21.15.103:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc304002080 con 0x7fc30c14be90 2024-03-25T19:25:43.816 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.816+0000 7fc2f37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:43.816 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.816+0000 7fc2f37fe640 1 -- 172.21.15.83:0/1971898268 <== mon.1 v1:172.21.15.103:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fc304003b10 con 0x7fc30c14be90 2024-03-25T19:25:43.816 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.816+0000 7fc2f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:43.816 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.816+0000 7fc2f37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:25:43.816 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.816+0000 7fc2f37fe640 10 monclient: dump: 2024-03-25T19:25:43.816 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:25:43.816 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:25:43.816 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:43.817 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:43.817 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:43.817 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:25:43.817 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:25:43.817 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:25:43.817 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:25:43.817 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:43.817 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.816+0000 7fc3097fa640 4 set_mon_vals no callback set 2024-03-25T19:25:43.818 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.817+0000 7fc2f37fe640 1 -- 172.21.15.83:0/1971898268 <== mon.1 v1:172.21.15.103:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7fc304003d30 con 0x7fc30c14be90 2024-03-25T19:25:43.818 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.818+0000 7fc30affd640 10 monclient: get_auth_request con 0x7fc2f806dfb0 auth_method 0 2024-03-25T19:25:43.818 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.818+0000 7fc2f37fe640 10 client.4501.objecter ms_handle_connect 0x7fc2f806dfb0 2024-03-25T19:25:43.822 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.823+0000 7fc2f37fe640 1 -- 172.21.15.83:0/1971898268 <== mon.1 v1:172.21.15.103:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7fc3040537c0 con 0x7fc30c14be90 2024-03-25T19:25:43.823 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.823+0000 7fc2f37fe640 10 client.4501.objecter ms_dispatch 0x7fc30c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:25:43.823 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.823+0000 7fc2f37fe640 3 client.4501.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:25:43.823 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.823+0000 7fc2f37fe640 3 client.4501.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:25:43.823 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.823+0000 7fc2f37fe640 20 client.4501.objecter dump_active .. 0 homeless 2024-03-25T19:25:43.823 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.823+0000 7fc2f37fe640 1 -- 172.21.15.83:0/1971898268 <== mon.1 v1:172.21.15.103:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc30404f2d0 con 0x7fc30c14be90 2024-03-25T19:25:43.823 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.823+0000 7fc2f37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:43.823 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:43.823+0000 7fc2f37fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:25:44.082 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:44.082+0000 7fc31178f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:44.082 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:44.082+0000 7fc31178f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:44.082 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:44.082+0000 7fc31178f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:25:44.082 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:44.082+0000 7fc31178f640 1 -- 172.21.15.83:0/1971898268 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc30c0b5cf0 con 0x7fc30c14be90 2024-03-25T19:25:44.118 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:44.119+0000 7fc2f37fe640 1 -- 172.21.15.83:0/1971898268 <== mon.1 v1:172.21.15.103:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7fc304051200 con 0x7fc30c14be90 2024-03-25T19:25:44.119 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:44.119+0000 7fc2f37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:44.119 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:44.119+0000 7fc2f37fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:25:44.119 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:25:44.119 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:25:44.121 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:44.121+0000 7fc31178f640 10 librados: watch_flush enter 2024-03-25T19:25:44.121 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:44.121+0000 7fc31178f640 10 librados: watch_flush exit 2024-03-25T19:25:44.121 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:44.121+0000 7fc31178f640 20 client.4501.objecter shutdown clearing up homeless session... 2024-03-25T19:25:44.121 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:44.121+0000 7fc31178f640 10 client.4501.objecter successfully canceled tick 2024-03-25T19:25:44.121 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:44.121+0000 7fc31178f640 1 -- 172.21.15.83:0/1971898268 >> v1:172.21.15.103:6800/345072327 conn(0x7fc2f806dfb0 legacy=0x7fc2f8070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:44.121 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:44.121+0000 7fc31178f640 10 monclient: shutdown 2024-03-25T19:25:44.122 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:44.121+0000 7fc31178f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:44.122 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:44.121+0000 7fc31178f640 1 -- 172.21.15.83:0/1971898268 >> v1:172.21.15.103:6789/0 conn(0x7fc30c14be90 legacy=0x7fc30c1e7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:44.122 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:44.121+0000 7fc31178f640 1 -- 172.21.15.83:0/1971898268 shutdown_connections 2024-03-25T19:25:44.122 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:44.121+0000 7fc31178f640 1 -- 172.21.15.83:0/1971898268 >> 172.21.15.83:0/1971898268 conn(0x7fc30c0617a0 msgr2=0x7fc30c0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:44.122 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:44.121+0000 7fc31178f640 1 -- 172.21.15.83:0/1971898268 shutdown_connections 2024-03-25T19:25:44.122 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:44.121+0000 7fc31178f640 1 -- 172.21.15.83:0/1971898268 wait complete. 2024-03-25T19:25:44.122 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:44.121+0000 7fc31178f640 1 librados: shutdown 2024-03-25T19:25:44.132 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:25:48.173 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:06:30.100 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:48.231 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:06:30.158 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:48.268 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:06:30.196 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:48.283 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:06:30.211 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:48.288 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:06:30.198 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:48.293 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:06:30.203 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:48.294 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:06:30.222 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:48.318 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:06:30.228 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:48.319 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:06:30.247 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:50.940 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:25:51.055 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.056+0000 7f6bae5c2640 10 monclient: get_monmap_and_config 2024-03-25T19:25:51.056 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.056+0000 7f6bae5c2640 10 monclient: build_initial_monmap 2024-03-25T19:25:51.056 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.056+0000 7f6bae5c2640 10 monclient: monmap: 2024-03-25T19:25:51.056 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:25:51.056 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:25:51.056 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:25:51.057179+0000 2024-03-25T19:25:51.056 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:25:51.057179+0000 2024-03-25T19:25:51.056 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:51.056 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:25:51.056 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:25:51.056 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:25:51.056 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:25:51.056 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:51.057 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.057+0000 7f6bae5c2640 1 Processor -- start 2024-03-25T19:25:51.058 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.057+0000 7f6bae5c2640 1 -- start start 2024-03-25T19:25:51.058 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.057+0000 7f6bae5c2640 10 monclient: init 2024-03-25T19:25:51.058 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.057+0000 7f6bae5c2640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:51.058 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.057+0000 7f6bae5c2640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:25:51.058 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.057+0000 7f6bae5c2640 10 monclient(hunting): picked mon.noname-b con 0x7f6ba80a98f0 addr v1:172.21.15.83:6790/0 2024-03-25T19:25:51.058 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.057+0000 7f6bae5c2640 10 monclient(hunting): picked mon.noname-a con 0x7f6ba813f710 addr v1:172.21.15.83:6789/0 2024-03-25T19:25:51.058 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.057+0000 7f6bae5c2640 10 monclient(hunting): picked mon.noname-c con 0x7f6ba81430c0 addr v1:172.21.15.103:6789/0 2024-03-25T19:25:51.058 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.057+0000 7f6bae5c2640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ba8142520 con 0x7f6ba813f710 2024-03-25T19:25:51.058 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.057+0000 7f6bae5c2640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ba80b28a0 con 0x7f6ba81430c0 2024-03-25T19:25:51.058 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.057+0000 7f6bae5c2640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ba80b2a30 con 0x7f6ba80a98f0 2024-03-25T19:25:51.058 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.057+0000 7f6bae5c2640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:51.058 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.057+0000 7f6bae5c2640 10 monclient(hunting): authenticate will time out at 6408.329441s 2024-03-25T19:25:51.066 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.067+0000 7f6ba77fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f6ba813f710 0x7f6ba81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:54606/0 (socket says 172.21.15.83:54606) 2024-03-25T19:25:51.066 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.067+0000 7f6ba77fe640 1 -- 172.21.15.83:0/2441040225 learned_addr learned my addr 172.21.15.83:0/2441040225 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:25:51.072 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.072+0000 7f6ba6ffd640 1 -- 172.21.15.83:0/2441040225 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 733821460 0 0) 0x7f6ba8142520 con 0x7f6ba813f710 2024-03-25T19:25:51.072 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.072+0000 7f6ba6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:25:51.072 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.072+0000 7f6ba6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:51.072 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.072+0000 7f6ba6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:51.072 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.072+0000 7f6ba6ffd640 10 monclient(hunting): my global_id is 4494 2024-03-25T19:25:51.072 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.072+0000 7f6ba6ffd640 1 -- 172.21.15.83:0/2441040225 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b90003520 con 0x7f6ba813f710 2024-03-25T19:25:51.077 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.077+0000 7f6ba6ffd640 1 -- 172.21.15.83:0/2441040225 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 104739337 0 0) 0x7f6b90003520 con 0x7f6ba813f710 2024-03-25T19:25:51.077 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.077+0000 7f6ba6ffd640 1 -- 172.21.15.83:0/2441040225 >> v1:172.21.15.83:6790/0 conn(0x7f6ba80a98f0 legacy=0x7f6ba80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:51.077 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.077+0000 7f6ba6ffd640 1 -- 172.21.15.83:0/2441040225 >> v1:172.21.15.103:6789/0 conn(0x7f6ba81430c0 legacy=0x7f6ba814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:51.077 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.077+0000 7f6ba6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:25:51.077 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.077+0000 7f6ba6ffd640 1 monclient: found mon.noname-a 2024-03-25T19:25:51.077 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.077+0000 7f6ba6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:51.077 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.077+0000 7f6ba6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.83:6789/0 2024-03-25T19:25:51.077 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.077+0000 7f6ba6ffd640 1 -- 172.21.15.83:0/2441040225 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ba80b44c0 con 0x7f6ba813f710 2024-03-25T19:25:51.077 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.077+0000 7f6ba6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:25:51.077 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.077+0000 7f6ba6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:25:51.077 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.077+0000 7f6ba6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:51.077 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.077+0000 7f6bae5c2640 5 monclient: authenticate success, global_id 4494 2024-03-25T19:25:51.077 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.077+0000 7f6bae5c2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:25:51.081 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.081+0000 7f6ba6ffd640 1 -- 172.21.15.83:0/2441040225 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f6b9c001690 con 0x7f6ba813f710 2024-03-25T19:25:51.081 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.081+0000 7f6ba6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:51.081 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.081+0000 7f6ba6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:25:51.081 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.081+0000 7f6ba6ffd640 10 monclient: dump: 2024-03-25T19:25:51.081 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:25:51.081 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:25:51.081 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:51.081 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:51.081 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:51.082 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:25:51.082 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:25:51.082 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:25:51.082 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:25:51.082 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:51.082 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.081+0000 7f6bae5c2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:25:51.106 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.106+0000 7f6ba6ffd640 1 -- 172.21.15.83:0/2441040225 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6b9c001920 con 0x7f6ba813f710 2024-03-25T19:25:51.106 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.106+0000 7f6ba6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:51.106 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.106+0000 7f6ba6ffd640 1 -- 172.21.15.83:0/2441040225 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f6b9c001c30 con 0x7f6ba813f710 2024-03-25T19:25:51.106 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.106+0000 7f6ba6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:51.106 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.106+0000 7f6ba6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:25:51.106 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.106+0000 7f6ba6ffd640 10 monclient: dump: 2024-03-25T19:25:51.106 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:25:51.106 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:25:51.107 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:51.107 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:51.107 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:51.107 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:25:51.107 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:25:51.107 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:25:51.107 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:25:51.107 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:51.107 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.106+0000 7f6bae5c2640 10 monclient: get_monmap_and_config success 2024-03-25T19:25:51.107 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.106+0000 7f6bae5c2640 4 set_mon_vals no callback set 2024-03-25T19:25:51.108 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.107+0000 7f6bae5c2640 10 monclient: shutdown 2024-03-25T19:25:51.108 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.107+0000 7f6bae5c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:51.108 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.107+0000 7f6bae5c2640 1 -- 172.21.15.83:0/2441040225 >> v1:172.21.15.83:6789/0 conn(0x7f6ba813f710 legacy=0x7f6ba81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:51.109 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.107+0000 7f6bae5c2640 1 -- 172.21.15.83:0/2441040225 shutdown_connections 2024-03-25T19:25:51.109 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.107+0000 7f6bae5c2640 1 -- 172.21.15.83:0/2441040225 >> 172.21.15.83:0/2441040225 conn(0x7f6ba80b7ea0 msgr2=0x7f6ba80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:51.109 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.107+0000 7f6bae5c2640 1 -- 172.21.15.83:0/2441040225 shutdown_connections 2024-03-25T19:25:51.109 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.107+0000 7f6bae5c2640 1 -- 172.21.15.83:0/2441040225 wait complete. 2024-03-25T19:25:51.109 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.108+0000 7f6bae5c2640 10 monclient: build_initial_monmap 2024-03-25T19:25:51.109 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.108+0000 7f6bae5c2640 10 monclient: monmap: 2024-03-25T19:25:51.109 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:25:51.109 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:25:51.109 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:25:51.109 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:25:51.109 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:51.109 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:25:51.109 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:25:51.109 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:25:51.109 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:25:51.110 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:51.110 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.108+0000 7f6bae5c2640 1 librados: starting msgr at 2024-03-25T19:25:51.110 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.108+0000 7f6bae5c2640 1 librados: starting objecter 2024-03-25T19:25:51.110 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.108+0000 7f6bae5c2640 1 Processor -- start 2024-03-25T19:25:51.110 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.109+0000 7f6bae5c2640 1 -- start start 2024-03-25T19:25:51.110 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.109+0000 7f6bae5c2640 1 librados: setting wanted keys 2024-03-25T19:25:51.110 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.109+0000 7f6bae5c2640 1 librados: calling monclient init 2024-03-25T19:25:51.110 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.109+0000 7f6bae5c2640 10 monclient: init 2024-03-25T19:25:51.111 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.110+0000 7f6bae5c2640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:51.111 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.110+0000 7f6bae5c2640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:25:51.111 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.110+0000 7f6bae5c2640 10 monclient(hunting): picked mon.noname-a con 0x7f6ba80a98f0 addr v1:172.21.15.83:6789/0 2024-03-25T19:25:51.111 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.110+0000 7f6bae5c2640 10 monclient(hunting): picked mon.noname-b con 0x7f6ba813f710 addr v1:172.21.15.103:6789/0 2024-03-25T19:25:51.111 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.110+0000 7f6bae5c2640 10 monclient(hunting): picked mon.noname-c con 0x7f6ba81430c0 addr v1:172.21.15.83:6790/0 2024-03-25T19:25:51.111 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.110+0000 7f6bae5c2640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ba8136f10 con 0x7f6ba80a98f0 2024-03-25T19:25:51.112 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.110+0000 7f6bae5c2640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ba81319b0 con 0x7f6ba813f710 2024-03-25T19:25:51.112 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.110+0000 7f6bae5c2640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ba81351a0 con 0x7f6ba81430c0 2024-03-25T19:25:51.112 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.110+0000 7f6bae5c2640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:51.112 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.110+0000 7f6bae5c2640 10 monclient(hunting): authenticate will time out at 6408.382138s 2024-03-25T19:25:51.116 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.117+0000 7f6ba7fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f6ba80a98f0 0x7f6ba81dab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:54638/0 (socket says 172.21.15.83:54638) 2024-03-25T19:25:51.116 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.117+0000 7f6ba7fff640 1 -- 172.21.15.83:0/1626842568 learned_addr learned my addr 172.21.15.83:0/1626842568 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:25:51.119 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.119+0000 7f6ba4ff9640 10 client.?.objecter ms_handle_connect 0x7f6ba80a98f0 2024-03-25T19:25:51.119 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.119+0000 7f6ba4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:51.121 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.122+0000 7f6ba4ff9640 10 client.?.objecter ms_handle_connect 0x7f6ba81430c0 2024-03-25T19:25:51.121 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.122+0000 7f6ba4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:51.126 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.126+0000 7f6ba4ff9640 1 -- 172.21.15.83:0/1626842568 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2508487520 0 0) 0x7f6ba8136f10 con 0x7f6ba80a98f0 2024-03-25T19:25:51.126 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.126+0000 7f6ba4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:25:51.126 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.126+0000 7f6ba4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:51.126 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.126+0000 7f6ba4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:51.126 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.126+0000 7f6ba4ff9640 10 monclient(hunting): my global_id is 4497 2024-03-25T19:25:51.126 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.126+0000 7f6ba4ff9640 1 -- 172.21.15.83:0/1626842568 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b94004eb0 con 0x7f6ba80a98f0 2024-03-25T19:25:51.133 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.131+0000 7f6ba4ff9640 10 client.?.objecter ms_handle_connect 0x7f6ba813f710 2024-03-25T19:25:51.133 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.131+0000 7f6ba4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:51.138 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.139+0000 7f6ba4ff9640 1 -- 172.21.15.83:0/1626842568 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 982679477 0 0) 0x7f6ba81351a0 con 0x7f6ba81430c0 2024-03-25T19:25:51.138 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.139+0000 7f6ba4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:25:51.139 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.139+0000 7f6ba4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:51.139 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.139+0000 7f6ba4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:51.139 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.139+0000 7f6ba4ff9640 10 monclient(hunting): my global_id is 4514 2024-03-25T19:25:51.139 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.139+0000 7f6ba4ff9640 1 -- 172.21.15.83:0/1626842568 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ba8136f10 con 0x7f6ba81430c0 2024-03-25T19:25:51.143 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.143+0000 7f6ba4ff9640 1 -- 172.21.15.83:0/1626842568 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1726248174 0 0) 0x7f6ba81319b0 con 0x7f6ba813f710 2024-03-25T19:25:51.143 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.143+0000 7f6ba4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:25:51.143 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.143+0000 7f6ba4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:51.143 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.143+0000 7f6ba4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:51.143 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.143+0000 7f6ba4ff9640 10 monclient(hunting): my global_id is 4507 2024-03-25T19:25:51.143 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.143+0000 7f6ba4ff9640 1 -- 172.21.15.83:0/1626842568 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ba81351a0 con 0x7f6ba813f710 2024-03-25T19:25:51.144 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.144+0000 7f6ba4ff9640 1 -- 172.21.15.83:0/1626842568 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1926675628 0 0) 0x7f6b94004eb0 con 0x7f6ba80a98f0 2024-03-25T19:25:51.144 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.144+0000 7f6ba4ff9640 1 -- 172.21.15.83:0/1626842568 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ba81319b0 con 0x7f6ba80a98f0 2024-03-25T19:25:51.144 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.144+0000 7f6ba4ff9640 1 -- 172.21.15.83:0/1626842568 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f6b980019d0 con 0x7f6ba80a98f0 2024-03-25T19:25:51.144 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.144+0000 7f6ba4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:51.144 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.144+0000 7f6ba4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:25:51.144 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.144+0000 7f6ba4ff9640 10 monclient(hunting): dump: 2024-03-25T19:25:51.144 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:25:51.144 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:25:51.144 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:51.144 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:51.144 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:51.144 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:25:51.145 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:25:51.145 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:25:51.145 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:25:51.145 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:51.145 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.144+0000 7f6ba4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:25:51.162 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.161+0000 7f6ba4ff9640 1 -- 172.21.15.83:0/1626842568 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1617322346 0 0) 0x7f6ba81319b0 con 0x7f6ba80a98f0 2024-03-25T19:25:51.162 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.161+0000 7f6ba4ff9640 1 -- 172.21.15.83:0/1626842568 >> v1:172.21.15.83:6790/0 conn(0x7f6ba81430c0 legacy=0x7f6ba81312a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:51.162 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.161+0000 7f6ba4ff9640 1 -- 172.21.15.83:0/1626842568 >> v1:172.21.15.103:6789/0 conn(0x7f6ba813f710 legacy=0x7f6ba8130b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:51.162 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.161+0000 7f6ba4ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:25:51.162 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.161+0000 7f6ba4ff9640 1 monclient: found mon.a 2024-03-25T19:25:51.162 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.161+0000 7f6ba4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:51.162 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.161+0000 7f6ba4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:25:51.162 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.161+0000 7f6ba4ff9640 1 -- 172.21.15.83:0/1626842568 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ba8135cb0 con 0x7f6ba80a98f0 2024-03-25T19:25:51.162 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.161+0000 7f6ba4ff9640 10 monclient: _finish_auth 0 2024-03-25T19:25:51.162 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.161+0000 7f6ba4ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:25:51.162 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.161+0000 7f6ba4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:51.162 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.161+0000 7f6bae5c2640 5 monclient: authenticate success, global_id 4497 2024-03-25T19:25:51.162 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.161+0000 7f6bae5c2640 10 monclient: _renew_subs 2024-03-25T19:25:51.163 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.161+0000 7f6bae5c2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:25:51.163 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.161+0000 7f6bae5c2640 1 -- 172.21.15.83:0/1626842568 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ba8135f00 con 0x7f6ba80a98f0 2024-03-25T19:25:51.163 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.161+0000 7f6bae5c2640 10 client.4497.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:25:51.163 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.161+0000 7f6bae5c2640 10 monclient: _renew_subs 2024-03-25T19:25:51.163 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.161+0000 7f6bae5c2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:25:51.163 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.161+0000 7f6bae5c2640 1 -- 172.21.15.83:0/1626842568 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ba813b9e0 con 0x7f6ba80a98f0 2024-03-25T19:25:51.163 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.161+0000 7f6bae5c2640 1 librados: init done 2024-03-25T19:25:51.163 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.162+0000 7f6bae5c2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:51.163 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.162+0000 7f6bae5c2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:51.164 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.162+0000 7f6bae5c2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:25:51.164 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.162+0000 7f6bae5c2640 1 -- 172.21.15.83:0/1626842568 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ba80627b0 con 0x7f6ba80a98f0 2024-03-25T19:25:51.456 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.456+0000 7f6ba4ff9640 1 -- 172.21.15.83:0/1626842568 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6b980035f0 con 0x7f6ba80a98f0 2024-03-25T19:25:51.456 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.456+0000 7f6ba4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:51.456 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.456+0000 7f6ba4ff9640 1 -- 172.21.15.83:0/1626842568 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f6b98003980 con 0x7f6ba80a98f0 2024-03-25T19:25:51.456 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.456+0000 7f6ba4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:51.456 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.456+0000 7f6ba4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:25:51.456 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.456+0000 7f6ba4ff9640 10 monclient: dump: 2024-03-25T19:25:51.456 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:25:51.456 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:25:51.456 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:51.457 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:51.457 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:51.457 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:25:51.457 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:25:51.457 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:25:51.457 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:25:51.457 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:51.457 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.456+0000 7f6ba67fc640 4 set_mon_vals no callback set 2024-03-25T19:25:51.458 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.457+0000 7f6ba4ff9640 1 -- 172.21.15.83:0/1626842568 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7f6b98003ba0 con 0x7f6ba80a98f0 2024-03-25T19:25:51.458 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.458+0000 7f6ba77fe640 10 monclient: get_auth_request con 0x7f6b9406ddf0 auth_method 0 2024-03-25T19:25:51.458 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.458+0000 7f6ba4ff9640 10 client.4497.objecter ms_handle_connect 0x7f6b9406ddf0 2024-03-25T19:25:51.462 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.462+0000 7f6ba4ff9640 1 -- 172.21.15.83:0/1626842568 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7f6b980547c0 con 0x7f6ba80a98f0 2024-03-25T19:25:51.463 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.462+0000 7f6ba4ff9640 10 client.4497.objecter ms_dispatch 0x7f6ba8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:25:51.463 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.462+0000 7f6ba4ff9640 3 client.4497.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:25:51.463 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.462+0000 7f6ba4ff9640 3 client.4497.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:25:51.463 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.462+0000 7f6ba4ff9640 20 client.4497.objecter dump_active .. 0 homeless 2024-03-25T19:25:51.463 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.462+0000 7f6ba4ff9640 1 -- 172.21.15.83:0/1626842568 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6b980502d0 con 0x7f6ba80a98f0 2024-03-25T19:25:51.463 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.462+0000 7f6ba4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:51.463 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.462+0000 7f6ba4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:25:51.713 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.714+0000 7f6bae5c2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:51.713 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.714+0000 7f6bae5c2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:51.713 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.714+0000 7f6bae5c2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:25:51.713 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.714+0000 7f6bae5c2640 1 -- 172.21.15.83:0/1626842568 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6ba8135330 con 0x7f6ba80a98f0 2024-03-25T19:25:51.793 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.793+0000 7f6ba4ff9640 1 -- 172.21.15.83:0/1626842568 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7f6b98052200 con 0x7f6ba80a98f0 2024-03-25T19:25:51.793 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.793+0000 7f6ba4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:51.793 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.793+0000 7f6ba4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:25:51.793 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:25:51.793 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:25:51.796 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.796+0000 7f6bae5c2640 10 librados: watch_flush enter 2024-03-25T19:25:51.796 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.796+0000 7f6bae5c2640 10 librados: watch_flush exit 2024-03-25T19:25:51.796 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.796+0000 7f6bae5c2640 20 client.4497.objecter shutdown clearing up homeless session... 2024-03-25T19:25:51.797 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.796+0000 7f6bae5c2640 10 client.4497.objecter successfully canceled tick 2024-03-25T19:25:51.797 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.796+0000 7f6bae5c2640 1 -- 172.21.15.83:0/1626842568 >> v1:172.21.15.103:6800/345072327 conn(0x7f6b9406ddf0 legacy=0x7f6b940702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:51.797 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.796+0000 7f6bae5c2640 10 monclient: shutdown 2024-03-25T19:25:51.797 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.796+0000 7f6bae5c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:51.797 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.796+0000 7f6bae5c2640 1 -- 172.21.15.83:0/1626842568 >> v1:172.21.15.83:6789/0 conn(0x7f6ba80a98f0 legacy=0x7f6ba81dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:51.797 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.796+0000 7f6bae5c2640 1 -- 172.21.15.83:0/1626842568 shutdown_connections 2024-03-25T19:25:51.797 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.796+0000 7f6bae5c2640 1 -- 172.21.15.83:0/1626842568 >> 172.21.15.83:0/1626842568 conn(0x7f6ba80b7ea0 msgr2=0x7f6ba813ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:51.797 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.796+0000 7f6bae5c2640 1 -- 172.21.15.83:0/1626842568 shutdown_connections 2024-03-25T19:25:51.797 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.796+0000 7f6bae5c2640 1 -- 172.21.15.83:0/1626842568 wait complete. 2024-03-25T19:25:51.797 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:51.796+0000 7f6bae5c2640 1 librados: shutdown 2024-03-25T19:25:51.809 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:25:58.615 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:25:58.731 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.731+0000 7f1489893640 10 monclient: get_monmap_and_config 2024-03-25T19:25:58.731 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.731+0000 7f1489893640 10 monclient: build_initial_monmap 2024-03-25T19:25:58.731 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.731+0000 7f1489893640 10 monclient: monmap: 2024-03-25T19:25:58.731 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:25:58.731 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:25:58.731 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:25:58.732357+0000 2024-03-25T19:25:58.731 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:25:58.732357+0000 2024-03-25T19:25:58.732 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:58.732 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:25:58.732 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:25:58.732 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:25:58.732 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:25:58.732 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:58.733 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.732+0000 7f1489893640 1 Processor -- start 2024-03-25T19:25:58.733 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.732+0000 7f1489893640 1 -- start start 2024-03-25T19:25:58.733 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.732+0000 7f1489893640 10 monclient: init 2024-03-25T19:25:58.733 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.733+0000 7f1489893640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:58.733 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.733+0000 7f1489893640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:25:58.733 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.733+0000 7f1489893640 10 monclient(hunting): picked mon.noname-b con 0x7f14840a98f0 addr v1:172.21.15.83:6790/0 2024-03-25T19:25:58.733 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.733+0000 7f1489893640 10 monclient(hunting): picked mon.noname-a con 0x7f148413f710 addr v1:172.21.15.83:6789/0 2024-03-25T19:25:58.733 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.733+0000 7f1489893640 10 monclient(hunting): picked mon.noname-c con 0x7f14841430c0 addr v1:172.21.15.103:6789/0 2024-03-25T19:25:58.733 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.733+0000 7f1489893640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1484142520 con 0x7f148413f710 2024-03-25T19:25:58.733 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.733+0000 7f1489893640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14840b28a0 con 0x7f14841430c0 2024-03-25T19:25:58.733 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.733+0000 7f1489893640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14840b2a30 con 0x7f14840a98f0 2024-03-25T19:25:58.734 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.733+0000 7f1489893640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:58.734 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.733+0000 7f1489893640 10 monclient(hunting): authenticate will time out at 6416.005056s 2024-03-25T19:25:58.739 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.740+0000 7f1482ffd640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f14840a98f0 0x7f14840a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:52734/0 (socket says 172.21.15.83:52734) 2024-03-25T19:25:58.739 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.740+0000 7f1482ffd640 1 -- 172.21.15.83:0/1431262512 learned_addr learned my addr 172.21.15.83:0/1431262512 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:25:58.756 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.757+0000 7f1481ffb640 1 -- 172.21.15.83:0/1431262512 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2239108728 0 0) 0x7f14840b2a30 con 0x7f14840a98f0 2024-03-25T19:25:58.756 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.757+0000 7f1481ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:25:58.756 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.757+0000 7f1481ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:58.756 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.757+0000 7f1481ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:58.756 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.757+0000 7f1481ffb640 10 monclient(hunting): my global_id is 4517 2024-03-25T19:25:58.757 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.757+0000 7f1481ffb640 1 -- 172.21.15.83:0/1431262512 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f146c003520 con 0x7f14840a98f0 2024-03-25T19:25:58.760 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.760+0000 7f1481ffb640 1 -- 172.21.15.83:0/1431262512 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1509932977 0 0) 0x7f14840b28a0 con 0x7f14841430c0 2024-03-25T19:25:58.760 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.760+0000 7f1481ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:25:58.760 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.760+0000 7f1481ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:58.760 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.760+0000 7f1481ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:58.761 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.760+0000 7f1481ffb640 10 monclient(hunting): my global_id is 4510 2024-03-25T19:25:58.761 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.760+0000 7f1481ffb640 1 -- 172.21.15.83:0/1431262512 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14840b2a30 con 0x7f14841430c0 2024-03-25T19:25:58.770 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.771+0000 7f1481ffb640 1 -- 172.21.15.83:0/1431262512 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1441970092 0 0) 0x7f1484142520 con 0x7f148413f710 2024-03-25T19:25:58.771 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.771+0000 7f1481ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:25:58.771 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.771+0000 7f1481ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:58.771 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.771+0000 7f1481ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:58.771 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.771+0000 7f1481ffb640 10 monclient(hunting): my global_id is 4500 2024-03-25T19:25:58.771 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.771+0000 7f1481ffb640 1 -- 172.21.15.83:0/1431262512 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14840b28a0 con 0x7f148413f710 2024-03-25T19:25:58.774 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.774+0000 7f1481ffb640 1 -- 172.21.15.83:0/1431262512 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3572598648 0 0) 0x7f146c003520 con 0x7f14840a98f0 2024-03-25T19:25:58.774 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.774+0000 7f1481ffb640 1 -- 172.21.15.83:0/1431262512 >> v1:172.21.15.103:6789/0 conn(0x7f14841430c0 legacy=0x7f148414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:58.774 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.774+0000 7f1481ffb640 1 -- 172.21.15.83:0/1431262512 >> v1:172.21.15.83:6789/0 conn(0x7f148413f710 legacy=0x7f14841429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:58.774 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.774+0000 7f1481ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:25:58.774 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.774+0000 7f1481ffb640 1 monclient: found mon.noname-b 2024-03-25T19:25:58.774 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.774+0000 7f1481ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:58.774 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.774+0000 7f1481ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.83:6790/0 2024-03-25T19:25:58.774 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.774+0000 7f1481ffb640 1 -- 172.21.15.83:0/1431262512 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14840b44c0 con 0x7f14840a98f0 2024-03-25T19:25:58.774 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.774+0000 7f1481ffb640 10 monclient: _finish_auth 0 2024-03-25T19:25:58.775 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.774+0000 7f1481ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:25:58.775 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.774+0000 7f1481ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:58.775 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.774+0000 7f1481ffb640 1 -- 172.21.15.83:0/1431262512 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f1474001690 con 0x7f14840a98f0 2024-03-25T19:25:58.775 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.774+0000 7f1481ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:58.775 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.774+0000 7f1481ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:25:58.775 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.774+0000 7f1481ffb640 10 monclient: dump: 2024-03-25T19:25:58.775 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:25:58.775 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:25:58.775 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:58.775 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:58.775 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:58.775 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:25:58.775 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:25:58.775 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:25:58.775 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:25:58.775 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:58.776 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.774+0000 7f1489893640 5 monclient: authenticate success, global_id 4517 2024-03-25T19:25:58.776 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.774+0000 7f1489893640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:25:58.784 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.784+0000 7f1481ffb640 1 -- 172.21.15.83:0/1431262512 <== mon.2 v1:172.21.15.83:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f14740017f0 con 0x7f14840a98f0 2024-03-25T19:25:58.784 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.784+0000 7f1481ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:58.784 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.784+0000 7f1481ffb640 1 -- 172.21.15.83:0/1431262512 <== mon.2 v1:172.21.15.83:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f1474001c00 con 0x7f14840a98f0 2024-03-25T19:25:58.784 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.784+0000 7f1481ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:58.784 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.784+0000 7f1481ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:25:58.784 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.784+0000 7f1481ffb640 10 monclient: dump: 2024-03-25T19:25:58.784 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:25:58.784 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:25:58.784 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:58.785 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:58.785 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:58.785 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:25:58.785 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:25:58.785 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:25:58.785 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:25:58.785 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:58.785 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.784+0000 7f1489893640 10 monclient: get_monmap_and_config success 2024-03-25T19:25:58.785 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.784+0000 7f1489893640 4 set_mon_vals no callback set 2024-03-25T19:25:58.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.785+0000 7f1489893640 10 monclient: shutdown 2024-03-25T19:25:58.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.785+0000 7f1489893640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:58.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.785+0000 7f1489893640 1 -- 172.21.15.83:0/1431262512 >> v1:172.21.15.83:6790/0 conn(0x7f14840a98f0 legacy=0x7f14840a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:58.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.785+0000 7f1489893640 1 -- 172.21.15.83:0/1431262512 shutdown_connections 2024-03-25T19:25:58.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.785+0000 7f1489893640 1 -- 172.21.15.83:0/1431262512 >> 172.21.15.83:0/1431262512 conn(0x7f14840b7ea0 msgr2=0x7f14840b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:58.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.785+0000 7f1489893640 1 -- 172.21.15.83:0/1431262512 shutdown_connections 2024-03-25T19:25:58.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.785+0000 7f1489893640 1 -- 172.21.15.83:0/1431262512 wait complete. 2024-03-25T19:25:58.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.786+0000 7f1489893640 10 monclient: build_initial_monmap 2024-03-25T19:25:58.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.786+0000 7f1489893640 10 monclient: monmap: 2024-03-25T19:25:58.787 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:25:58.787 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:25:58.787 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:25:58.787 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:25:58.787 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:58.788 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:25:58.788 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:25:58.788 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:25:58.788 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:25:58.788 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:58.788 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.787+0000 7f1489893640 1 librados: starting msgr at 2024-03-25T19:25:58.788 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.787+0000 7f1489893640 1 librados: starting objecter 2024-03-25T19:25:58.788 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.787+0000 7f1489893640 1 Processor -- start 2024-03-25T19:25:58.788 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.787+0000 7f1489893640 1 -- start start 2024-03-25T19:25:58.788 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.787+0000 7f1489893640 1 librados: setting wanted keys 2024-03-25T19:25:58.788 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.787+0000 7f1489893640 1 librados: calling monclient init 2024-03-25T19:25:58.788 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.787+0000 7f1489893640 10 monclient: init 2024-03-25T19:25:58.789 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.788+0000 7f1489893640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:58.789 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.788+0000 7f1489893640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:25:58.790 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.788+0000 7f1489893640 10 monclient(hunting): picked mon.noname-a con 0x7f14840a98f0 addr v1:172.21.15.83:6789/0 2024-03-25T19:25:58.790 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.788+0000 7f1489893640 10 monclient(hunting): picked mon.noname-c con 0x7f148413f710 addr v1:172.21.15.83:6790/0 2024-03-25T19:25:58.790 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.788+0000 7f1489893640 10 monclient(hunting): picked mon.noname-b con 0x7f14841430c0 addr v1:172.21.15.103:6789/0 2024-03-25T19:25:58.790 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.788+0000 7f1489893640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1484135610 con 0x7f14840a98f0 2024-03-25T19:25:58.790 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.788+0000 7f1489893640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1484137230 con 0x7f14841430c0 2024-03-25T19:25:58.790 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.788+0000 7f1489893640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1484135cf0 con 0x7f148413f710 2024-03-25T19:25:58.790 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.788+0000 7f1489893640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:58.790 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.788+0000 7f1489893640 10 monclient(hunting): authenticate will time out at 6416.060542s 2024-03-25T19:25:58.795 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.795+0000 7f14837fe640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f14841430c0 0x7f14841e2ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:48638/0 (socket says 172.21.15.83:48638) 2024-03-25T19:25:58.795 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.795+0000 7f14837fe640 1 -- 172.21.15.83:0/152686905 learned_addr learned my addr 172.21.15.83:0/152686905 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:25:58.796 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.797+0000 7f146bfff640 10 client.?.objecter ms_handle_connect 0x7f14841430c0 2024-03-25T19:25:58.796 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.797+0000 7f146bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:58.812 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.812+0000 7f146bfff640 1 -- 172.21.15.83:0/152686905 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3094091768 0 0) 0x7f1484137230 con 0x7f14841430c0 2024-03-25T19:25:58.812 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.812+0000 7f146bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:25:58.812 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.812+0000 7f146bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:58.812 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.812+0000 7f146bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:58.812 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.812+0000 7f146bfff640 10 monclient(hunting): my global_id is 4513 2024-03-25T19:25:58.812 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.812+0000 7f146bfff640 1 -- 172.21.15.83:0/152686905 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1470004eb0 con 0x7f14841430c0 2024-03-25T19:25:58.814 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.815+0000 7f146bfff640 10 client.?.objecter ms_handle_connect 0x7f148413f710 2024-03-25T19:25:58.814 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.815+0000 7f146bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:58.818 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.818+0000 7f146bfff640 10 client.?.objecter ms_handle_connect 0x7f14840a98f0 2024-03-25T19:25:58.818 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.818+0000 7f146bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:58.831 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.831+0000 7f146bfff640 1 -- 172.21.15.83:0/152686905 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1319259612 0 0) 0x7f1484135610 con 0x7f14840a98f0 2024-03-25T19:25:58.831 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.831+0000 7f146bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:25:58.831 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.831+0000 7f146bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:58.831 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.831+0000 7f146bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:58.831 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.831+0000 7f146bfff640 10 monclient(hunting): my global_id is 4503 2024-03-25T19:25:58.831 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.831+0000 7f146bfff640 1 -- 172.21.15.83:0/152686905 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1484137230 con 0x7f14840a98f0 2024-03-25T19:25:58.833 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.834+0000 7f146bfff640 1 -- 172.21.15.83:0/152686905 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 122790620 0 0) 0x7f1470004eb0 con 0x7f14841430c0 2024-03-25T19:25:58.834 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.834+0000 7f146bfff640 1 -- 172.21.15.83:0/152686905 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1484135610 con 0x7f14841430c0 2024-03-25T19:25:58.834 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.834+0000 7f146bfff640 1 -- 172.21.15.83:0/152686905 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f147c001c90 con 0x7f14841430c0 2024-03-25T19:25:58.834 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.834+0000 7f146bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:58.834 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.834+0000 7f146bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:25:58.834 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.834+0000 7f146bfff640 10 monclient(hunting): dump: 2024-03-25T19:25:58.834 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:25:58.834 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:25:58.834 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:58.834 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:58.834 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:58.834 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:25:58.834 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:25:58.834 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:25:58.834 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:25:58.835 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:58.835 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.834+0000 7f146bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:25:58.835 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.835+0000 7f146bfff640 1 -- 172.21.15.83:0/152686905 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3425855605 0 0) 0x7f1484135cf0 con 0x7f148413f710 2024-03-25T19:25:58.835 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.835+0000 7f146bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:25:58.835 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.835+0000 7f146bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:58.835 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.835+0000 7f146bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:58.835 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.835+0000 7f146bfff640 10 monclient(hunting): my global_id is 4520 2024-03-25T19:25:58.835 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.835+0000 7f146bfff640 1 -- 172.21.15.83:0/152686905 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1470004eb0 con 0x7f148413f710 2024-03-25T19:25:58.840 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.840+0000 7f146bfff640 1 -- 172.21.15.83:0/152686905 <== mon.1 v1:172.21.15.103:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3202885870 0 0) 0x7f1484135610 con 0x7f14841430c0 2024-03-25T19:25:58.841 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.840+0000 7f146bfff640 1 -- 172.21.15.83:0/152686905 >> v1:172.21.15.83:6790/0 conn(0x7f148413f710 legacy=0x7f14841dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:58.841 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.840+0000 7f146bfff640 1 -- 172.21.15.83:0/152686905 >> v1:172.21.15.83:6789/0 conn(0x7f14840a98f0 legacy=0x7f1484130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:58.841 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.840+0000 7f146bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:25:58.841 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.840+0000 7f146bfff640 1 monclient: found mon.b 2024-03-25T19:25:58.841 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.840+0000 7f146bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:58.841 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.840+0000 7f146bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:25:58.841 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.840+0000 7f146bfff640 1 -- 172.21.15.83:0/152686905 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1484135e80 con 0x7f14841430c0 2024-03-25T19:25:58.841 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.840+0000 7f146bfff640 10 monclient: _finish_auth 0 2024-03-25T19:25:58.841 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.840+0000 7f146bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:25:58.841 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.840+0000 7f146bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:58.841 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.840+0000 7f1489893640 5 monclient: authenticate success, global_id 4513 2024-03-25T19:25:58.841 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.840+0000 7f1489893640 10 monclient: _renew_subs 2024-03-25T19:25:58.841 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.840+0000 7f1489893640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:25:58.841 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.840+0000 7f1489893640 1 -- 172.21.15.83:0/152686905 --> v1:172.21.15.103:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1484131660 con 0x7f14841430c0 2024-03-25T19:25:58.842 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.840+0000 7f1489893640 10 client.4513.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:25:58.842 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.840+0000 7f1489893640 10 monclient: _renew_subs 2024-03-25T19:25:58.842 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.840+0000 7f1489893640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:25:58.842 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.840+0000 7f1489893640 1 -- 172.21.15.83:0/152686905 --> v1:172.21.15.103:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f14840b7960 con 0x7f14841430c0 2024-03-25T19:25:58.842 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.840+0000 7f1489893640 1 librados: init done 2024-03-25T19:25:58.842 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.842+0000 7f1489893640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:58.842 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.842+0000 7f1489893640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:58.843 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.842+0000 7f1489893640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:25:58.843 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.842+0000 7f1489893640 1 -- 172.21.15.83:0/152686905 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14840627b0 con 0x7f14841430c0 2024-03-25T19:25:58.845 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.845+0000 7f146bfff640 1 -- 172.21.15.83:0/152686905 <== mon.1 v1:172.21.15.103:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f147c003970 con 0x7f14841430c0 2024-03-25T19:25:58.845 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.845+0000 7f146bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:58.845 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:58.845+0000 7f14817fa640 4 set_mon_vals no callback set 2024-03-25T19:25:59.128 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:59.128+0000 7f146bfff640 1 -- 172.21.15.83:0/152686905 <== mon.1 v1:172.21.15.103:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f147c003d40 con 0x7f14841430c0 2024-03-25T19:25:59.128 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:59.128+0000 7f146bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:59.128 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:59.128+0000 7f146bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:25:59.128 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:59.128+0000 7f146bfff640 10 monclient: dump: 2024-03-25T19:25:59.128 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:25:59.128 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:25:59.128 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:59.128 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:25:59.128 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:59.128 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:25:59.129 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:25:59.129 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:25:59.129 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:25:59.129 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:25:59.129 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:59.130+0000 7f146bfff640 1 -- 172.21.15.83:0/152686905 <== mon.1 v1:172.21.15.103:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7f147c003f20 con 0x7f14841430c0 2024-03-25T19:25:59.130 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:59.130+0000 7f1482ffd640 10 monclient: get_auth_request con 0x7f147006de00 auth_method 0 2024-03-25T19:25:59.130 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:59.131+0000 7f146bfff640 10 client.4513.objecter ms_handle_connect 0x7f147006de00 2024-03-25T19:25:59.132 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:59.132+0000 7f146bfff640 1 -- 172.21.15.83:0/152686905 <== mon.1 v1:172.21.15.103:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7f147c054b50 con 0x7f14841430c0 2024-03-25T19:25:59.132 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:59.132+0000 7f146bfff640 10 client.4513.objecter ms_dispatch 0x7f1484000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:25:59.132 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:59.132+0000 7f146bfff640 3 client.4513.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:25:59.132 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:59.132+0000 7f146bfff640 3 client.4513.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:25:59.132 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:59.133+0000 7f146bfff640 20 client.4513.objecter dump_active .. 0 homeless 2024-03-25T19:25:59.135 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:59.136+0000 7f146bfff640 1 -- 172.21.15.83:0/152686905 <== mon.1 v1:172.21.15.103:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f147c050660 con 0x7f14841430c0 2024-03-25T19:25:59.135 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:59.136+0000 7f146bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:59.135 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:59.136+0000 7f146bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:25:59.359 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:59.359+0000 7f1489893640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:59.359 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:59.359+0000 7f1489893640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:59.359 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:59.359+0000 7f1489893640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:25:59.359 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:59.359+0000 7f1489893640 1 -- 172.21.15.83:0/152686905 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f148414f240 con 0x7f14841430c0 2024-03-25T19:25:59.384 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:59.384+0000 7f146bfff640 1 -- 172.21.15.83:0/152686905 <== mon.1 v1:172.21.15.103:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7f147c052590 con 0x7f14841430c0 2024-03-25T19:25:59.384 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:59.385+0000 7f146bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:59.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:59.385+0000 7f146bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:25:59.385 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:25:59.385 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:25:59.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:59.387+0000 7f1489893640 10 librados: watch_flush enter 2024-03-25T19:25:59.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:59.387+0000 7f1489893640 10 librados: watch_flush exit 2024-03-25T19:25:59.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:59.387+0000 7f1489893640 20 client.4513.objecter shutdown clearing up homeless session... 2024-03-25T19:25:59.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:59.387+0000 7f1489893640 10 client.4513.objecter successfully canceled tick 2024-03-25T19:25:59.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:59.387+0000 7f1489893640 1 -- 172.21.15.83:0/152686905 >> v1:172.21.15.103:6800/345072327 conn(0x7f147006de00 legacy=0x7f14700702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:59.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:59.387+0000 7f1489893640 10 monclient: shutdown 2024-03-25T19:25:59.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:59.387+0000 7f1489893640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:59.388 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:59.387+0000 7f1489893640 1 -- 172.21.15.83:0/152686905 >> v1:172.21.15.103:6789/0 conn(0x7f14841430c0 legacy=0x7f14841e2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:59.388 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:59.387+0000 7f1489893640 1 -- 172.21.15.83:0/152686905 shutdown_connections 2024-03-25T19:25:59.388 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:59.387+0000 7f1489893640 1 -- 172.21.15.83:0/152686905 >> 172.21.15.83:0/152686905 conn(0x7f14840b7ea0 msgr2=0x7f1484074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:59.388 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:59.387+0000 7f1489893640 1 -- 172.21.15.83:0/152686905 shutdown_connections 2024-03-25T19:25:59.388 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:59.387+0000 7f1489893640 1 -- 172.21.15.83:0/152686905 wait complete. 2024-03-25T19:25:59.388 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:25:59.387+0000 7f1489893640 1 librados: shutdown 2024-03-25T19:25:59.399 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:26:06.205 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:26:06.305 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.305+0000 7f85b5b89640 10 monclient: get_monmap_and_config 2024-03-25T19:26:06.305 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.305+0000 7f85b5b89640 10 monclient: build_initial_monmap 2024-03-25T19:26:06.305 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.305+0000 7f85b5b89640 10 monclient: monmap: 2024-03-25T19:26:06.305 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:26:06.305 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:26:06.305 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:26:06.306593+0000 2024-03-25T19:26:06.305 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:26:06.306593+0000 2024-03-25T19:26:06.305 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:26:06.306 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:26:06.306 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:26:06.306 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:26:06.306 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:26:06.306 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:06.307 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.306+0000 7f85b5b89640 1 Processor -- start 2024-03-25T19:26:06.307 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.306+0000 7f85b5b89640 1 -- start start 2024-03-25T19:26:06.307 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.306+0000 7f85b5b89640 10 monclient: init 2024-03-25T19:26:06.307 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.307+0000 7f85b5b89640 10 monclient: _reopen_session rank -1 2024-03-25T19:26:06.307 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.307+0000 7f85b5b89640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:26:06.307 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.307+0000 7f85b5b89640 10 monclient(hunting): picked mon.noname-b con 0x7f85b00b4d60 addr v1:172.21.15.83:6790/0 2024-03-25T19:26:06.307 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.307+0000 7f85b5b89640 10 monclient(hunting): picked mon.noname-c con 0x7f85b00b1730 addr v1:172.21.15.103:6789/0 2024-03-25T19:26:06.307 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.307+0000 7f85b5b89640 10 monclient(hunting): picked mon.noname-a con 0x7f85b014be90 addr v1:172.21.15.83:6789/0 2024-03-25T19:26:06.308 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.307+0000 7f85b5b89640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85b00af940 con 0x7f85b014be90 2024-03-25T19:26:06.308 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.307+0000 7f85b5b89640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85b00b0a70 con 0x7f85b00b1730 2024-03-25T19:26:06.308 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.307+0000 7f85b5b89640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85b00b0d50 con 0x7f85b00b4d60 2024-03-25T19:26:06.308 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.307+0000 7f85b5b89640 10 monclient(hunting): _renew_subs 2024-03-25T19:26:06.308 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.307+0000 7f85b5b89640 10 monclient(hunting): authenticate will time out at 6423.579360s 2024-03-25T19:26:06.310 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.311+0000 7f85aeffd640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f85b00b1730 0x7f85b00b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:59928/0 (socket says 172.21.15.83:59928) 2024-03-25T19:26:06.310 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.311+0000 7f85aeffd640 1 -- 172.21.15.83:0/3986067973 learned_addr learned my addr 172.21.15.83:0/3986067973 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:26:06.319 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.320+0000 7f85ae7fc640 1 -- 172.21.15.83:0/3986067973 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3308193347 0 0) 0x7f85b00af940 con 0x7f85b014be90 2024-03-25T19:26:06.319 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.320+0000 7f85ae7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:26:06.319 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.320+0000 7f85ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:06.319 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.320+0000 7f85ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:06.320 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.320+0000 7f85ae7fc640 10 monclient(hunting): my global_id is 4506 2024-03-25T19:26:06.320 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.320+0000 7f85ae7fc640 1 -- 172.21.15.83:0/3986067973 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8598003520 con 0x7f85b014be90 2024-03-25T19:26:06.325 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.326+0000 7f85ae7fc640 1 -- 172.21.15.83:0/3986067973 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2840991705 0 0) 0x7f85b00b0a70 con 0x7f85b00b1730 2024-03-25T19:26:06.325 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.326+0000 7f85ae7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:26:06.325 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.326+0000 7f85ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:06.326 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.326+0000 7f85ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:06.326 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.326+0000 7f85ae7fc640 10 monclient(hunting): my global_id is 4516 2024-03-25T19:26:06.326 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.326+0000 7f85ae7fc640 1 -- 172.21.15.83:0/3986067973 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85b00af940 con 0x7f85b00b1730 2024-03-25T19:26:06.337 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.337+0000 7f85ae7fc640 1 -- 172.21.15.83:0/3986067973 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1012570979 0 0) 0x7f8598003520 con 0x7f85b014be90 2024-03-25T19:26:06.337 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.337+0000 7f85ae7fc640 1 -- 172.21.15.83:0/3986067973 >> v1:172.21.15.83:6790/0 conn(0x7f85b00b4d60 legacy=0x7f85b013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:06.337 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.337+0000 7f85ae7fc640 1 -- 172.21.15.83:0/3986067973 >> v1:172.21.15.103:6789/0 conn(0x7f85b00b1730 legacy=0x7f85b00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:06.337 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.337+0000 7f85ae7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:26:06.338 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.337+0000 7f85ae7fc640 1 monclient: found mon.noname-a 2024-03-25T19:26:06.338 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.337+0000 7f85ae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:26:06.338 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.337+0000 7f85ae7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.83:6789/0 2024-03-25T19:26:06.338 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.337+0000 7f85ae7fc640 1 -- 172.21.15.83:0/3986067973 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85b00b1030 con 0x7f85b014be90 2024-03-25T19:26:06.338 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.337+0000 7f85ae7fc640 10 monclient: _finish_auth 0 2024-03-25T19:26:06.338 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.337+0000 7f85ae7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:26:06.338 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.337+0000 7f85ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:26:06.338 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.337+0000 7f85ae7fc640 1 -- 172.21.15.83:0/3986067973 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f85a8001a10 con 0x7f85b014be90 2024-03-25T19:26:06.338 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.337+0000 7f85ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:06.338 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.337+0000 7f85ae7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:26:06.338 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.337+0000 7f85ae7fc640 10 monclient: dump: 2024-03-25T19:26:06.338 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:26:06.338 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:26:06.338 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:06.338 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:06.339 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:06.339 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:26:06.339 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:26:06.339 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:26:06.339 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:26:06.339 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:06.339 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.337+0000 7f85b5b89640 5 monclient: authenticate success, global_id 4506 2024-03-25T19:26:06.339 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.337+0000 7f85b5b89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:26:06.352 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.352+0000 7f85ae7fc640 1 -- 172.21.15.83:0/3986067973 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85a8001c30 con 0x7f85b014be90 2024-03-25T19:26:06.352 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.352+0000 7f85ae7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:26:06.352 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.352+0000 7f85ae7fc640 1 -- 172.21.15.83:0/3986067973 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f85a8001fc0 con 0x7f85b014be90 2024-03-25T19:26:06.352 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.352+0000 7f85b5b89640 10 monclient: get_monmap_and_config success 2024-03-25T19:26:06.352 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.352+0000 7f85b5b89640 4 set_mon_vals no callback set 2024-03-25T19:26:06.353 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.352+0000 7f85ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:06.353 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.352+0000 7f85ae7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:26:06.353 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.352+0000 7f85ae7fc640 10 monclient: dump: 2024-03-25T19:26:06.353 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:26:06.353 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:26:06.353 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:06.353 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:06.353 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:06.353 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:26:06.353 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:26:06.353 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:26:06.353 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:26:06.353 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:06.354 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.352+0000 7f85b5b89640 10 monclient: shutdown 2024-03-25T19:26:06.354 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.352+0000 7f85b5b89640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:26:06.354 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.352+0000 7f85b5b89640 1 -- 172.21.15.83:0/3986067973 >> v1:172.21.15.83:6789/0 conn(0x7f85b014be90 legacy=0x7f85b014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:06.354 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.352+0000 7f85b5b89640 1 -- 172.21.15.83:0/3986067973 shutdown_connections 2024-03-25T19:26:06.354 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.352+0000 7f85b5b89640 1 -- 172.21.15.83:0/3986067973 >> 172.21.15.83:0/3986067973 conn(0x7f85b00617a0 msgr2=0x7f85b0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:26:06.354 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.352+0000 7f85b5b89640 1 -- 172.21.15.83:0/3986067973 shutdown_connections 2024-03-25T19:26:06.354 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.352+0000 7f85b5b89640 1 -- 172.21.15.83:0/3986067973 wait complete. 2024-03-25T19:26:06.356 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.353+0000 7f85b5b89640 10 monclient: build_initial_monmap 2024-03-25T19:26:06.356 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.353+0000 7f85b5b89640 10 monclient: monmap: 2024-03-25T19:26:06.356 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:26:06.356 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:26:06.356 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:26:06.356 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:26:06.356 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:26:06.356 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:26:06.356 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:26:06.356 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:26:06.356 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:26:06.357 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:06.357 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.354+0000 7f85b5b89640 1 librados: starting msgr at 2024-03-25T19:26:06.357 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.354+0000 7f85b5b89640 1 librados: starting objecter 2024-03-25T19:26:06.357 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.354+0000 7f85b5b89640 1 Processor -- start 2024-03-25T19:26:06.357 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.354+0000 7f85b5b89640 1 -- start start 2024-03-25T19:26:06.357 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.354+0000 7f85b5b89640 1 librados: setting wanted keys 2024-03-25T19:26:06.357 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.354+0000 7f85b5b89640 1 librados: calling monclient init 2024-03-25T19:26:06.357 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.354+0000 7f85b5b89640 10 monclient: init 2024-03-25T19:26:06.357 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.355+0000 7f85b5b89640 10 monclient: _reopen_session rank -1 2024-03-25T19:26:06.357 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.355+0000 7f85b5b89640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:26:06.357 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.355+0000 7f85b5b89640 10 monclient(hunting): picked mon.noname-c con 0x7f85b00b1730 addr v1:172.21.15.83:6790/0 2024-03-25T19:26:06.357 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.355+0000 7f85b5b89640 10 monclient(hunting): picked mon.noname-b con 0x7f85b00b4d60 addr v1:172.21.15.103:6789/0 2024-03-25T19:26:06.357 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.355+0000 7f85b5b89640 10 monclient(hunting): picked mon.noname-a con 0x7f85b014be90 addr v1:172.21.15.83:6789/0 2024-03-25T19:26:06.357 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.355+0000 7f85b5b89640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85b01d86f0 con 0x7f85b014be90 2024-03-25T19:26:06.357 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.355+0000 7f85b5b89640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85b013b4f0 con 0x7f85b00b4d60 2024-03-25T19:26:06.358 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.355+0000 7f85b5b89640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85b01d7eb0 con 0x7f85b00b1730 2024-03-25T19:26:06.358 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.355+0000 7f85b5b89640 10 monclient(hunting): _renew_subs 2024-03-25T19:26:06.358 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.355+0000 7f85b5b89640 10 monclient(hunting): authenticate will time out at 6423.627203s 2024-03-25T19:26:06.358 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.358+0000 7f85af7fe640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f85b00b1730 0x7f85b01d77a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:59910/0 (socket says 172.21.15.83:59910) 2024-03-25T19:26:06.358 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.358+0000 7f85af7fe640 1 -- 172.21.15.83:0/3641892124 learned_addr learned my addr 172.21.15.83:0/3641892124 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:26:06.360 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.360+0000 7f8597fff640 10 client.?.objecter ms_handle_connect 0x7f85b00b1730 2024-03-25T19:26:06.360 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.360+0000 7f8597fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:06.360 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.360+0000 7f8597fff640 10 client.?.objecter ms_handle_connect 0x7f85b00b4d60 2024-03-25T19:26:06.360 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.360+0000 7f8597fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:06.365 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.365+0000 7f8597fff640 1 -- 172.21.15.83:0/3641892124 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3786872259 0 0) 0x7f85b01d7eb0 con 0x7f85b00b1730 2024-03-25T19:26:06.365 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.365+0000 7f8597fff640 10 monclient(hunting): _negotiate 2024-03-25T19:26:06.365 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.365+0000 7f8597fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:06.366 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.365+0000 7f8597fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:06.366 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.365+0000 7f8597fff640 10 monclient(hunting): my global_id is 4526 2024-03-25T19:26:06.366 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.365+0000 7f8597fff640 1 -- 172.21.15.83:0/3641892124 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f859c004eb0 con 0x7f85b00b1730 2024-03-25T19:26:06.366 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.366+0000 7f8597fff640 1 -- 172.21.15.83:0/3641892124 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3816107506 0 0) 0x7f85b013b4f0 con 0x7f85b00b4d60 2024-03-25T19:26:06.366 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.366+0000 7f8597fff640 10 monclient(hunting): _negotiate 2024-03-25T19:26:06.366 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.366+0000 7f8597fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:06.366 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.366+0000 7f8597fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:06.366 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.366+0000 7f8597fff640 10 monclient(hunting): my global_id is 4519 2024-03-25T19:26:06.366 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.366+0000 7f8597fff640 1 -- 172.21.15.83:0/3641892124 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85b01d7eb0 con 0x7f85b00b4d60 2024-03-25T19:26:06.368 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.368+0000 7f8597fff640 10 client.?.objecter ms_handle_connect 0x7f85b014be90 2024-03-25T19:26:06.368 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.368+0000 7f8597fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:06.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.371+0000 7f8597fff640 1 -- 172.21.15.83:0/3641892124 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3927103044 0 0) 0x7f859c004eb0 con 0x7f85b00b1730 2024-03-25T19:26:06.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.371+0000 7f8597fff640 1 -- 172.21.15.83:0/3641892124 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f85b013b4f0 con 0x7f85b00b1730 2024-03-25T19:26:06.374 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.374+0000 7f8597fff640 1 -- 172.21.15.83:0/3641892124 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 640820794 0 0) 0x7f85b00b1030 con 0x7f85b014be90 2024-03-25T19:26:06.374 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.374+0000 7f8597fff640 10 monclient(hunting): _negotiate 2024-03-25T19:26:06.374 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.374+0000 7f8597fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:06.374 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.374+0000 7f8597fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:06.374 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.374+0000 7f8597fff640 10 monclient(hunting): my global_id is 4509 2024-03-25T19:26:06.374 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.374+0000 7f8597fff640 1 -- 172.21.15.83:0/3641892124 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f859c004eb0 con 0x7f85b014be90 2024-03-25T19:26:06.375 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.375+0000 7f8597fff640 1 -- 172.21.15.83:0/3641892124 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f85a0001ad0 con 0x7f85b00b1730 2024-03-25T19:26:06.375 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.375+0000 7f8597fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:06.375 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.375+0000 7f8597fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:26:06.375 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.375+0000 7f8597fff640 10 monclient(hunting): dump: 2024-03-25T19:26:06.375 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:26:06.375 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:26:06.375 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:06.376 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:06.376 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:06.376 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:26:06.376 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:26:06.376 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:26:06.376 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:26:06.376 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:06.376 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.375+0000 7f8597fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:26:06.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.385+0000 7f8597fff640 1 -- 172.21.15.83:0/3641892124 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3324805842 0 0) 0x7f85b01d7eb0 con 0x7f85b00b4d60 2024-03-25T19:26:06.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.385+0000 7f8597fff640 1 -- 172.21.15.83:0/3641892124 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f85b00b1030 con 0x7f85b00b4d60 2024-03-25T19:26:06.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.385+0000 7f8597fff640 1 -- 172.21.15.83:0/3641892124 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f85a4001930 con 0x7f85b00b4d60 2024-03-25T19:26:06.386 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.385+0000 7f8597fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:06.386 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.385+0000 7f8597fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:26:06.386 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.385+0000 7f8597fff640 10 monclient(hunting): dump: 2024-03-25T19:26:06.386 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:26:06.386 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:26:06.386 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:06.386 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:06.386 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:06.386 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:26:06.386 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:26:06.386 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:26:06.386 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:26:06.386 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:06.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.385+0000 7f8597fff640 1 -- 172.21.15.83:0/3641892124 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3748124704 0 0) 0x7f85b013b4f0 con 0x7f85b00b1730 2024-03-25T19:26:06.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.385+0000 7f8597fff640 1 -- 172.21.15.83:0/3641892124 >> v1:172.21.15.103:6789/0 conn(0x7f85b00b4d60 legacy=0x7f85b0136fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:06.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.385+0000 7f8597fff640 1 -- 172.21.15.83:0/3641892124 >> v1:172.21.15.83:6789/0 conn(0x7f85b014be90 legacy=0x7f85b01e3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:06.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.385+0000 7f8597fff640 10 monclient: _finish_hunting 0 2024-03-25T19:26:06.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.385+0000 7f8597fff640 1 monclient: found mon.c 2024-03-25T19:26:06.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.385+0000 7f8597fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:26:06.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.385+0000 7f8597fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:26:06.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.385+0000 7f8597fff640 1 -- 172.21.15.83:0/3641892124 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85b01d8040 con 0x7f85b00b1730 2024-03-25T19:26:06.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.385+0000 7f8597fff640 10 monclient: _finish_auth 0 2024-03-25T19:26:06.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.385+0000 7f8597fff640 10 monclient: _check_auth_tickets 2024-03-25T19:26:06.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.385+0000 7f8597fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:26:06.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.385+0000 7f85b5b89640 5 monclient: authenticate success, global_id 4526 2024-03-25T19:26:06.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.385+0000 7f85b5b89640 10 monclient: _renew_subs 2024-03-25T19:26:06.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.385+0000 7f85b5b89640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:26:06.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.385+0000 7f85b5b89640 1 -- 172.21.15.83:0/3641892124 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85b013ba70 con 0x7f85b00b1730 2024-03-25T19:26:06.388 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.385+0000 7f85b5b89640 10 client.4526.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:26:06.388 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.385+0000 7f85b5b89640 10 monclient: _renew_subs 2024-03-25T19:26:06.388 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.385+0000 7f85b5b89640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:26:06.388 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.385+0000 7f85b5b89640 1 -- 172.21.15.83:0/3641892124 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f85b01378d0 con 0x7f85b00b1730 2024-03-25T19:26:06.388 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.385+0000 7f85b5b89640 1 librados: init done 2024-03-25T19:26:06.388 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.386+0000 7f85b5b89640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:06.388 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.386+0000 7f85b5b89640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:06.388 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.386+0000 7f85b5b89640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:26:06.389 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.386+0000 7f85b5b89640 1 -- 172.21.15.83:0/3641892124 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85b00627b0 con 0x7f85b00b1730 2024-03-25T19:26:06.690 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.690+0000 7f8597fff640 1 -- 172.21.15.83:0/3641892124 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85a0001c30 con 0x7f85b00b1730 2024-03-25T19:26:06.690 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.690+0000 7f8597fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:26:06.691 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.690+0000 7f8597fff640 1 -- 172.21.15.83:0/3641892124 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f85a0003a60 con 0x7f85b00b1730 2024-03-25T19:26:06.691 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.690+0000 7f8597fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:06.691 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.690+0000 7f8597fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:26:06.691 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.690+0000 7f8597fff640 10 monclient: dump: 2024-03-25T19:26:06.691 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:26:06.691 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:26:06.691 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:06.691 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:06.691 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:06.691 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:26:06.691 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:26:06.691 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:26:06.692 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:26:06.692 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:06.692 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.690+0000 7f85adffb640 4 set_mon_vals no callback set 2024-03-25T19:26:06.692 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.691+0000 7f8597fff640 1 -- 172.21.15.83:0/3641892124 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7f85a0003c80 con 0x7f85b00b1730 2024-03-25T19:26:06.692 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.692+0000 7f85aeffd640 10 monclient: get_auth_request con 0x7f859c06df70 auth_method 0 2024-03-25T19:26:06.692 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.692+0000 7f8597fff640 10 client.4526.objecter ms_handle_connect 0x7f859c06df70 2024-03-25T19:26:06.699 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.699+0000 7f8597fff640 1 -- 172.21.15.83:0/3641892124 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7f85a00548a0 con 0x7f85b00b1730 2024-03-25T19:26:06.699 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.699+0000 7f8597fff640 10 client.4526.objecter ms_dispatch 0x7f85b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:26:06.699 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.699+0000 7f8597fff640 3 client.4526.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:26:06.699 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.699+0000 7f8597fff640 3 client.4526.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:26:06.699 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.699+0000 7f8597fff640 20 client.4526.objecter dump_active .. 0 homeless 2024-03-25T19:26:06.699 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.699+0000 7f8597fff640 1 -- 172.21.15.83:0/3641892124 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f85a00503b0 con 0x7f85b00b1730 2024-03-25T19:26:06.699 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.699+0000 7f8597fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:06.699 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.699+0000 7f8597fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:26:06.944 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.944+0000 7f85b5b89640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:06.944 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.944+0000 7f85b5b89640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:06.944 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.944+0000 7f85b5b89640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:26:06.944 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.944+0000 7f85b5b89640 1 -- 172.21.15.83:0/3641892124 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f85b0069f40 con 0x7f85b00b1730 2024-03-25T19:26:06.966 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.966+0000 7f8597fff640 1 -- 172.21.15.83:0/3641892124 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7f85a00522e0 con 0x7f85b00b1730 2024-03-25T19:26:06.966 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.966+0000 7f8597fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:06.966 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.966+0000 7f8597fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:26:06.966 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:26:06.966 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:26:06.969 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.969+0000 7f85b5b89640 10 librados: watch_flush enter 2024-03-25T19:26:06.969 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.969+0000 7f85b5b89640 10 librados: watch_flush exit 2024-03-25T19:26:06.969 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.969+0000 7f85b5b89640 20 client.4526.objecter shutdown clearing up homeless session... 2024-03-25T19:26:06.969 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.969+0000 7f85b5b89640 10 client.4526.objecter successfully canceled tick 2024-03-25T19:26:06.969 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.969+0000 7f85b5b89640 1 -- 172.21.15.83:0/3641892124 >> v1:172.21.15.103:6800/345072327 conn(0x7f859c06df70 legacy=0x7f859c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:06.969 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.969+0000 7f85b5b89640 10 monclient: shutdown 2024-03-25T19:26:06.969 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.969+0000 7f85b5b89640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:26:06.969 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.969+0000 7f85b5b89640 1 -- 172.21.15.83:0/3641892124 >> v1:172.21.15.83:6790/0 conn(0x7f85b00b1730 legacy=0x7f85b01d77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:06.970 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.969+0000 7f85b5b89640 1 -- 172.21.15.83:0/3641892124 shutdown_connections 2024-03-25T19:26:06.970 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.969+0000 7f85b5b89640 1 -- 172.21.15.83:0/3641892124 >> 172.21.15.83:0/3641892124 conn(0x7f85b00617a0 msgr2=0x7f85b013c730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:26:06.970 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.969+0000 7f85b5b89640 1 -- 172.21.15.83:0/3641892124 shutdown_connections 2024-03-25T19:26:06.970 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.969+0000 7f85b5b89640 1 -- 172.21.15.83:0/3641892124 wait complete. 2024-03-25T19:26:06.970 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:06.969+0000 7f85b5b89640 1 librados: shutdown 2024-03-25T19:26:06.981 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:26:08.269 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:06:50.196 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:08.274 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:06:50.201 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:08.321 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:06:50.249 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:08.330 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:06:50.241 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:08.333 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:06:50.243 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:08.348 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:06:50.270 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:08.351 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:06:50.279 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:08.362 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:06:50.272 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:08.377 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:06:50.304 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:13.787 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:26:13.903 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.903+0000 7fbb4446a640 10 monclient: get_monmap_and_config 2024-03-25T19:26:13.904 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.903+0000 7fbb4446a640 10 monclient: build_initial_monmap 2024-03-25T19:26:13.904 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.904+0000 7fbb4446a640 10 monclient: monmap: 2024-03-25T19:26:13.904 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:26:13.904 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:26:13.904 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:26:13.904903+0000 2024-03-25T19:26:13.904 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:26:13.904903+0000 2024-03-25T19:26:13.904 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:26:13.904 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:26:13.904 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:26:13.904 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:26:13.904 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:26:13.904 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:13.905 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.904+0000 7fbb4446a640 1 Processor -- start 2024-03-25T19:26:13.906 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.904+0000 7fbb4446a640 1 -- start start 2024-03-25T19:26:13.906 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.904+0000 7fbb4446a640 10 monclient: init 2024-03-25T19:26:13.906 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.905+0000 7fbb4446a640 10 monclient: _reopen_session rank -1 2024-03-25T19:26:13.906 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.905+0000 7fbb4446a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:26:13.906 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.905+0000 7fbb4446a640 10 monclient(hunting): picked mon.noname-a con 0x7fbb3c0b4d60 addr v1:172.21.15.83:6789/0 2024-03-25T19:26:13.906 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.905+0000 7fbb4446a640 10 monclient(hunting): picked mon.noname-b con 0x7fbb3c0b1730 addr v1:172.21.15.83:6790/0 2024-03-25T19:26:13.906 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.905+0000 7fbb4446a640 10 monclient(hunting): picked mon.noname-c con 0x7fbb3c14be90 addr v1:172.21.15.103:6789/0 2024-03-25T19:26:13.906 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.905+0000 7fbb4446a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb3c0af940 con 0x7fbb3c0b4d60 2024-03-25T19:26:13.906 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.905+0000 7fbb4446a640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb3c0b0a70 con 0x7fbb3c14be90 2024-03-25T19:26:13.906 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.905+0000 7fbb4446a640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb3c0b0d50 con 0x7fbb3c0b1730 2024-03-25T19:26:13.906 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.905+0000 7fbb4446a640 10 monclient(hunting): _renew_subs 2024-03-25T19:26:13.906 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.905+0000 7fbb4446a640 10 monclient(hunting): authenticate will time out at 6431.177484s 2024-03-25T19:26:13.912 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.913+0000 7fbb42206640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fbb3c0b4d60 0x7fbb3c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:36344/0 (socket says 172.21.15.83:36344) 2024-03-25T19:26:13.912 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.913+0000 7fbb42206640 1 -- 172.21.15.83:0/2029634455 learned_addr learned my addr 172.21.15.83:0/2029634455 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:26:13.926 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.926+0000 7fbb41204640 1 -- 172.21.15.83:0/2029634455 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3264686701 0 0) 0x7fbb3c0b0a70 con 0x7fbb3c14be90 2024-03-25T19:26:13.926 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.926+0000 7fbb41204640 10 monclient(hunting): _negotiate 2024-03-25T19:26:13.926 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.926+0000 7fbb41204640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:13.926 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.926+0000 7fbb41204640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:13.926 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.926+0000 7fbb41204640 10 monclient(hunting): my global_id is 4522 2024-03-25T19:26:13.926 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.926+0000 7fbb41204640 1 -- 172.21.15.83:0/2029634455 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb28003520 con 0x7fbb3c14be90 2024-03-25T19:26:13.932 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.932+0000 7fbb41204640 1 -- 172.21.15.83:0/2029634455 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3024832997 0 0) 0x7fbb3c0af940 con 0x7fbb3c0b4d60 2024-03-25T19:26:13.932 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.932+0000 7fbb41204640 10 monclient(hunting): _negotiate 2024-03-25T19:26:13.932 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.932+0000 7fbb41204640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:13.932 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.932+0000 7fbb41204640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:13.932 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.932+0000 7fbb41204640 10 monclient(hunting): my global_id is 4512 2024-03-25T19:26:13.932 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.932+0000 7fbb41204640 1 -- 172.21.15.83:0/2029634455 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb3c0b0a70 con 0x7fbb3c0b4d60 2024-03-25T19:26:13.936 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.936+0000 7fbb41204640 1 -- 172.21.15.83:0/2029634455 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3370981522 0 0) 0x7fbb3c0b0d50 con 0x7fbb3c0b1730 2024-03-25T19:26:13.936 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.936+0000 7fbb41204640 10 monclient(hunting): _negotiate 2024-03-25T19:26:13.936 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.936+0000 7fbb41204640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:13.936 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.936+0000 7fbb41204640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:13.936 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.936+0000 7fbb41204640 10 monclient(hunting): my global_id is 4529 2024-03-25T19:26:13.936 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.936+0000 7fbb41204640 1 -- 172.21.15.83:0/2029634455 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb3c0af940 con 0x7fbb3c0b1730 2024-03-25T19:26:13.942 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.942+0000 7fbb41204640 1 -- 172.21.15.83:0/2029634455 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1774036967 0 0) 0x7fbb28003520 con 0x7fbb3c14be90 2024-03-25T19:26:13.943 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.942+0000 7fbb41204640 1 -- 172.21.15.83:0/2029634455 >> v1:172.21.15.83:6790/0 conn(0x7fbb3c0b1730 legacy=0x7fbb3c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:13.943 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.942+0000 7fbb41204640 1 -- 172.21.15.83:0/2029634455 >> v1:172.21.15.83:6789/0 conn(0x7fbb3c0b4d60 legacy=0x7fbb3c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:13.943 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.942+0000 7fbb41204640 10 monclient: _finish_hunting 0 2024-03-25T19:26:13.943 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.942+0000 7fbb41204640 1 monclient: found mon.noname-c 2024-03-25T19:26:13.943 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.942+0000 7fbb41204640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:26:13.943 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.942+0000 7fbb41204640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.103:6789/0 2024-03-25T19:26:13.943 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.942+0000 7fbb41204640 1 -- 172.21.15.83:0/2029634455 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb3c0b1030 con 0x7fbb3c14be90 2024-03-25T19:26:13.943 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.942+0000 7fbb41204640 10 monclient: _finish_auth 0 2024-03-25T19:26:13.943 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.942+0000 7fbb41204640 10 monclient: _check_auth_tickets 2024-03-25T19:26:13.943 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.942+0000 7fbb41204640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:26:13.943 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.942+0000 7fbb41204640 1 -- 172.21.15.83:0/2029634455 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fbb34001a10 con 0x7fbb3c14be90 2024-03-25T19:26:13.943 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.942+0000 7fbb41204640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:13.943 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.942+0000 7fbb41204640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:26:13.944 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.942+0000 7fbb41204640 10 monclient: dump: 2024-03-25T19:26:13.944 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:26:13.944 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:26:13.944 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:13.944 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:13.944 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:13.944 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:26:13.944 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:26:13.944 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:26:13.944 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:26:13.944 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:13.944 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.942+0000 7fbb4446a640 5 monclient: authenticate success, global_id 4522 2024-03-25T19:26:13.944 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.942+0000 7fbb4446a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:26:13.952 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.952+0000 7fbb41204640 1 -- 172.21.15.83:0/2029634455 <== mon.1 v1:172.21.15.103:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbb34001b70 con 0x7fbb3c14be90 2024-03-25T19:26:13.952 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.952+0000 7fbb41204640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:26:13.952 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.952+0000 7fbb41204640 1 -- 172.21.15.83:0/2029634455 <== mon.1 v1:172.21.15.103:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fbb34001f80 con 0x7fbb3c14be90 2024-03-25T19:26:13.952 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.952+0000 7fbb4446a640 10 monclient: get_monmap_and_config success 2024-03-25T19:26:13.952 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.952+0000 7fbb4446a640 4 set_mon_vals no callback set 2024-03-25T19:26:13.952 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.952+0000 7fbb41204640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:13.952 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.952+0000 7fbb41204640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:26:13.952 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.952+0000 7fbb41204640 10 monclient: dump: 2024-03-25T19:26:13.952 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:26:13.952 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:26:13.952 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:13.953 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:13.953 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:13.953 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:26:13.953 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:26:13.953 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:26:13.953 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:26:13.953 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:13.955 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.953+0000 7fbb4446a640 10 monclient: shutdown 2024-03-25T19:26:13.955 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.953+0000 7fbb4446a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:26:13.955 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.953+0000 7fbb4446a640 1 -- 172.21.15.83:0/2029634455 >> v1:172.21.15.103:6789/0 conn(0x7fbb3c14be90 legacy=0x7fbb3c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:13.955 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.953+0000 7fbb4446a640 1 -- 172.21.15.83:0/2029634455 shutdown_connections 2024-03-25T19:26:13.955 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.953+0000 7fbb4446a640 1 -- 172.21.15.83:0/2029634455 >> 172.21.15.83:0/2029634455 conn(0x7fbb3c0617a0 msgr2=0x7fbb3c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:26:13.955 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.953+0000 7fbb4446a640 1 -- 172.21.15.83:0/2029634455 shutdown_connections 2024-03-25T19:26:13.955 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.953+0000 7fbb4446a640 1 -- 172.21.15.83:0/2029634455 wait complete. 2024-03-25T19:26:13.955 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.954+0000 7fbb4446a640 10 monclient: build_initial_monmap 2024-03-25T19:26:13.955 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.954+0000 7fbb4446a640 10 monclient: monmap: 2024-03-25T19:26:13.955 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:26:13.955 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:26:13.955 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:26:13.956 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:26:13.956 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:26:13.956 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:26:13.956 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:26:13.956 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:26:13.956 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:26:13.956 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:13.956 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.954+0000 7fbb4446a640 1 librados: starting msgr at 2024-03-25T19:26:13.956 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.954+0000 7fbb4446a640 1 librados: starting objecter 2024-03-25T19:26:13.956 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.954+0000 7fbb4446a640 1 Processor -- start 2024-03-25T19:26:13.956 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.954+0000 7fbb4446a640 1 -- start start 2024-03-25T19:26:13.956 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.954+0000 7fbb4446a640 1 librados: setting wanted keys 2024-03-25T19:26:13.956 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.954+0000 7fbb4446a640 1 librados: calling monclient init 2024-03-25T19:26:13.956 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.954+0000 7fbb4446a640 10 monclient: init 2024-03-25T19:26:13.956 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.954+0000 7fbb4446a640 10 monclient: _reopen_session rank -1 2024-03-25T19:26:13.956 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.954+0000 7fbb4446a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:26:13.956 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.954+0000 7fbb4446a640 10 monclient(hunting): picked mon.noname-a con 0x7fbb3c0b1730 addr v1:172.21.15.83:6789/0 2024-03-25T19:26:13.956 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.955+0000 7fbb4446a640 10 monclient(hunting): picked mon.noname-c con 0x7fbb3c0b4d60 addr v1:172.21.15.83:6790/0 2024-03-25T19:26:13.957 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.955+0000 7fbb4446a640 10 monclient(hunting): picked mon.noname-b con 0x7fbb3c14be90 addr v1:172.21.15.103:6789/0 2024-03-25T19:26:13.957 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.955+0000 7fbb4446a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb3c138640 con 0x7fbb3c0b1730 2024-03-25T19:26:13.957 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.955+0000 7fbb4446a640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb3c139800 con 0x7fbb3c14be90 2024-03-25T19:26:13.957 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.955+0000 7fbb4446a640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb3c132620 con 0x7fbb3c0b4d60 2024-03-25T19:26:13.957 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.955+0000 7fbb4446a640 10 monclient(hunting): _renew_subs 2024-03-25T19:26:13.957 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.955+0000 7fbb4446a640 10 monclient(hunting): authenticate will time out at 6431.226877s 2024-03-25T19:26:13.962 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.962+0000 7fbb41a05640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fbb3c0b4d60 0x7fbb3c13b040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:36096/0 (socket says 172.21.15.83:36096) 2024-03-25T19:26:13.962 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.962+0000 7fbb41a05640 1 -- 172.21.15.83:0/1474170385 learned_addr learned my addr 172.21.15.83:0/1474170385 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:26:13.962 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.963+0000 7fbb32ffd640 10 client.?.objecter ms_handle_connect 0x7fbb3c0b4d60 2024-03-25T19:26:13.962 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.963+0000 7fbb32ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:13.965 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.965+0000 7fbb32ffd640 10 client.?.objecter ms_handle_connect 0x7fbb3c14be90 2024-03-25T19:26:13.965 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.965+0000 7fbb32ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:13.966 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.967+0000 7fbb32ffd640 10 client.?.objecter ms_handle_connect 0x7fbb3c0b1730 2024-03-25T19:26:13.966 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.967+0000 7fbb32ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:13.970 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.970+0000 7fbb32ffd640 1 -- 172.21.15.83:0/1474170385 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 855187659 0 0) 0x7fbb3c132620 con 0x7fbb3c0b4d60 2024-03-25T19:26:13.970 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.970+0000 7fbb32ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:26:13.970 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.970+0000 7fbb32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:13.970 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.970+0000 7fbb32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:13.970 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.970+0000 7fbb32ffd640 10 monclient(hunting): my global_id is 4532 2024-03-25T19:26:13.970 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.970+0000 7fbb32ffd640 1 -- 172.21.15.83:0/1474170385 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb20004eb0 con 0x7fbb3c0b4d60 2024-03-25T19:26:13.972 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.973+0000 7fbb32ffd640 1 -- 172.21.15.83:0/1474170385 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1280565557 0 0) 0x7fbb3c138640 con 0x7fbb3c0b1730 2024-03-25T19:26:13.973 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.973+0000 7fbb32ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:26:13.973 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.973+0000 7fbb32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:13.973 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.973+0000 7fbb32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:13.973 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.973+0000 7fbb32ffd640 10 monclient(hunting): my global_id is 4515 2024-03-25T19:26:13.973 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.973+0000 7fbb32ffd640 1 -- 172.21.15.83:0/1474170385 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb3c132620 con 0x7fbb3c0b1730 2024-03-25T19:26:13.982 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.982+0000 7fbb32ffd640 1 -- 172.21.15.83:0/1474170385 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1816716208 0 0) 0x7fbb3c139800 con 0x7fbb3c14be90 2024-03-25T19:26:13.982 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.982+0000 7fbb32ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:26:13.982 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.982+0000 7fbb32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:13.982 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.982+0000 7fbb32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:13.982 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.982+0000 7fbb32ffd640 10 monclient(hunting): my global_id is 4525 2024-03-25T19:26:13.982 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.982+0000 7fbb32ffd640 1 -- 172.21.15.83:0/1474170385 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb3c138640 con 0x7fbb3c14be90 2024-03-25T19:26:13.983 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.983+0000 7fbb32ffd640 1 -- 172.21.15.83:0/1474170385 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2984766504 0 0) 0x7fbb20004eb0 con 0x7fbb3c0b4d60 2024-03-25T19:26:13.983 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.983+0000 7fbb32ffd640 1 -- 172.21.15.83:0/1474170385 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb3c139800 con 0x7fbb3c0b4d60 2024-03-25T19:26:13.983 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.983+0000 7fbb32ffd640 1 -- 172.21.15.83:0/1474170385 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fbb2c001a10 con 0x7fbb3c0b4d60 2024-03-25T19:26:13.983 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.983+0000 7fbb32ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:13.983 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.983+0000 7fbb32ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:26:13.983 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.983+0000 7fbb32ffd640 10 monclient(hunting): dump: 2024-03-25T19:26:13.983 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:26:13.983 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:26:13.983 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:13.983 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:13.983 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:13.983 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:26:13.983 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:26:13.983 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:26:13.984 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:26:13.984 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:13.984 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.983+0000 7fbb32ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:26:13.985 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.985+0000 7fbb32ffd640 1 -- 172.21.15.83:0/1474170385 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3754507767 0 0) 0x7fbb3c132620 con 0x7fbb3c0b1730 2024-03-25T19:26:13.985 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.985+0000 7fbb32ffd640 1 -- 172.21.15.83:0/1474170385 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb20004eb0 con 0x7fbb3c0b1730 2024-03-25T19:26:13.985 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.985+0000 7fbb32ffd640 1 -- 172.21.15.83:0/1474170385 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fbb38001970 con 0x7fbb3c0b1730 2024-03-25T19:26:13.985 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.985+0000 7fbb32ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:13.985 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.985+0000 7fbb32ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:26:13.986 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.985+0000 7fbb32ffd640 10 monclient(hunting): dump: 2024-03-25T19:26:13.986 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:26:13.986 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:26:13.986 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:13.986 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:13.986 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:13.986 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:26:13.986 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:26:13.986 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:26:13.986 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:26:13.986 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:13.992 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.991+0000 7fbb32ffd640 1 -- 172.21.15.83:0/1474170385 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3212457981 0 0) 0x7fbb3c139800 con 0x7fbb3c0b4d60 2024-03-25T19:26:13.992 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.992+0000 7fbb32ffd640 1 -- 172.21.15.83:0/1474170385 >> v1:172.21.15.103:6789/0 conn(0x7fbb3c14be90 legacy=0x7fbb3c13b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:13.992 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.992+0000 7fbb32ffd640 1 -- 172.21.15.83:0/1474170385 >> v1:172.21.15.83:6789/0 conn(0x7fbb3c0b1730 legacy=0x7fbb3c131f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:13.992 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.992+0000 7fbb32ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:26:13.992 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.992+0000 7fbb32ffd640 1 monclient: found mon.c 2024-03-25T19:26:13.992 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.992+0000 7fbb32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:26:13.992 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.992+0000 7fbb32ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:26:13.992 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.992+0000 7fbb32ffd640 1 -- 172.21.15.83:0/1474170385 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb3c1327b0 con 0x7fbb3c0b4d60 2024-03-25T19:26:13.992 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.992+0000 7fbb32ffd640 10 monclient: _finish_auth 0 2024-03-25T19:26:13.992 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.992+0000 7fbb32ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:26:13.992 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.992+0000 7fbb32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:26:13.992 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.992+0000 7fbb4446a640 5 monclient: authenticate success, global_id 4532 2024-03-25T19:26:13.992 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.992+0000 7fbb4446a640 10 monclient: _renew_subs 2024-03-25T19:26:13.992 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.992+0000 7fbb4446a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:26:13.992 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.992+0000 7fbb4446a640 1 -- 172.21.15.83:0/1474170385 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb3c138dd0 con 0x7fbb3c0b4d60 2024-03-25T19:26:13.993 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.992+0000 7fbb4446a640 10 client.4532.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:26:13.993 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.992+0000 7fbb4446a640 10 monclient: _renew_subs 2024-03-25T19:26:13.993 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.992+0000 7fbb4446a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:26:13.993 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.992+0000 7fbb4446a640 1 -- 172.21.15.83:0/1474170385 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb3c138270 con 0x7fbb3c0b4d60 2024-03-25T19:26:13.993 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.992+0000 7fbb4446a640 1 librados: init done 2024-03-25T19:26:13.994 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.993+0000 7fbb4446a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:13.994 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.993+0000 7fbb4446a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:13.994 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.993+0000 7fbb4446a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:26:13.994 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:13.993+0000 7fbb4446a640 1 -- 172.21.15.83:0/1474170385 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb3c138fc0 con 0x7fbb3c0b4d60 2024-03-25T19:26:14.294 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:14.294+0000 7fbb32ffd640 1 -- 172.21.15.83:0/1474170385 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbb2c001c70 con 0x7fbb3c0b4d60 2024-03-25T19:26:14.294 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:14.294+0000 7fbb32ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:26:14.294 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:14.294+0000 7fbb32ffd640 1 -- 172.21.15.83:0/1474170385 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fbb2c003a60 con 0x7fbb3c0b4d60 2024-03-25T19:26:14.294 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:14.294+0000 7fbb32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:14.294 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:14.294+0000 7fbb32ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:26:14.294 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:14.294+0000 7fbb32ffd640 10 monclient: dump: 2024-03-25T19:26:14.294 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:26:14.294 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:26:14.294 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:14.295 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:14.295 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:14.295 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:26:14.295 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:26:14.295 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:26:14.295 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:26:14.295 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:14.295 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:14.294+0000 7fbb40a03640 4 set_mon_vals no callback set 2024-03-25T19:26:14.296 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:14.295+0000 7fbb32ffd640 1 -- 172.21.15.83:0/1474170385 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7fbb2c003c80 con 0x7fbb3c0b4d60 2024-03-25T19:26:14.296 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:14.295+0000 7fbb42206640 10 monclient: get_auth_request con 0x7fbb2006df70 auth_method 0 2024-03-25T19:26:14.296 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:14.296+0000 7fbb32ffd640 10 client.4532.objecter ms_handle_connect 0x7fbb2006df70 2024-03-25T19:26:14.300 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:14.300+0000 7fbb32ffd640 1 -- 172.21.15.83:0/1474170385 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7fbb2c054870 con 0x7fbb3c0b4d60 2024-03-25T19:26:14.300 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:14.300+0000 7fbb32ffd640 10 client.4532.objecter ms_dispatch 0x7fbb3c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:26:14.300 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:14.300+0000 7fbb32ffd640 3 client.4532.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:26:14.300 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:14.300+0000 7fbb32ffd640 3 client.4532.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:26:14.300 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:14.300+0000 7fbb32ffd640 20 client.4532.objecter dump_active .. 0 homeless 2024-03-25T19:26:14.300 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:14.300+0000 7fbb32ffd640 1 -- 172.21.15.83:0/1474170385 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbb2c050380 con 0x7fbb3c0b4d60 2024-03-25T19:26:14.300 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:14.300+0000 7fbb32ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:14.300 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:14.300+0000 7fbb32ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:26:14.533 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:14.533+0000 7fbb4446a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:14.533 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:14.533+0000 7fbb4446a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:14.533 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:14.533+0000 7fbb4446a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:26:14.533 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:14.533+0000 7fbb4446a640 1 -- 172.21.15.83:0/1474170385 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbb3c0b2270 con 0x7fbb3c0b4d60 2024-03-25T19:26:14.576 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:14.576+0000 7fbb32ffd640 1 -- 172.21.15.83:0/1474170385 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7fbb2c0522b0 con 0x7fbb3c0b4d60 2024-03-25T19:26:14.576 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:14.576+0000 7fbb32ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:14.576 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:14.576+0000 7fbb32ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:26:14.577 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:26:14.577 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:26:14.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:14.579+0000 7fbb4446a640 10 librados: watch_flush enter 2024-03-25T19:26:14.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:14.579+0000 7fbb4446a640 10 librados: watch_flush exit 2024-03-25T19:26:14.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:14.579+0000 7fbb4446a640 20 client.4532.objecter shutdown clearing up homeless session... 2024-03-25T19:26:14.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:14.579+0000 7fbb4446a640 10 client.4532.objecter successfully canceled tick 2024-03-25T19:26:14.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:14.579+0000 7fbb4446a640 1 -- 172.21.15.83:0/1474170385 >> v1:172.21.15.103:6800/345072327 conn(0x7fbb2006df70 legacy=0x7fbb20070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:14.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:14.579+0000 7fbb4446a640 10 monclient: shutdown 2024-03-25T19:26:14.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:14.579+0000 7fbb4446a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:26:14.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:14.579+0000 7fbb4446a640 1 -- 172.21.15.83:0/1474170385 >> v1:172.21.15.83:6790/0 conn(0x7fbb3c0b4d60 legacy=0x7fbb3c13b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:14.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:14.579+0000 7fbb4446a640 1 -- 172.21.15.83:0/1474170385 shutdown_connections 2024-03-25T19:26:14.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:14.579+0000 7fbb4446a640 1 -- 172.21.15.83:0/1474170385 >> 172.21.15.83:0/1474170385 conn(0x7fbb3c0617a0 msgr2=0x7fbb3c13c450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:26:14.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:14.580+0000 7fbb4446a640 1 -- 172.21.15.83:0/1474170385 shutdown_connections 2024-03-25T19:26:14.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:14.580+0000 7fbb4446a640 1 -- 172.21.15.83:0/1474170385 wait complete. 2024-03-25T19:26:14.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:14.580+0000 7fbb4446a640 1 librados: shutdown 2024-03-25T19:26:14.591 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:26:21.398 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:26:21.512 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.513+0000 7fbc09520640 10 monclient: get_monmap_and_config 2024-03-25T19:26:21.513 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.513+0000 7fbc09520640 10 monclient: build_initial_monmap 2024-03-25T19:26:21.513 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.513+0000 7fbc09520640 10 monclient: monmap: 2024-03-25T19:26:21.513 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:26:21.513 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:26:21.513 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:26:21.514114+0000 2024-03-25T19:26:21.513 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:26:21.514114+0000 2024-03-25T19:26:21.513 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:26:21.513 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:26:21.513 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:26:21.513 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:26:21.513 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:26:21.513 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:21.515 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.514+0000 7fbc09520640 1 Processor -- start 2024-03-25T19:26:21.515 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.514+0000 7fbc09520640 1 -- start start 2024-03-25T19:26:21.515 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.514+0000 7fbc09520640 10 monclient: init 2024-03-25T19:26:21.515 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.515+0000 7fbc09520640 10 monclient: _reopen_session rank -1 2024-03-25T19:26:21.515 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.515+0000 7fbc09520640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:26:21.515 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.515+0000 7fbc09520640 10 monclient(hunting): picked mon.noname-b con 0x7fbc04061830 addr v1:172.21.15.83:6790/0 2024-03-25T19:26:21.515 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.515+0000 7fbc09520640 10 monclient(hunting): picked mon.noname-c con 0x7fbc0406ff10 addr v1:172.21.15.103:6789/0 2024-03-25T19:26:21.515 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.515+0000 7fbc09520640 10 monclient(hunting): picked mon.noname-a con 0x7fbc0413c950 addr v1:172.21.15.83:6789/0 2024-03-25T19:26:21.515 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.515+0000 7fbc09520640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc04071a60 con 0x7fbc0413c950 2024-03-25T19:26:21.515 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.515+0000 7fbc09520640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc04070bb0 con 0x7fbc0406ff10 2024-03-25T19:26:21.515 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.515+0000 7fbc09520640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc04071cd0 con 0x7fbc04061830 2024-03-25T19:26:21.515 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.515+0000 7fbc09520640 10 monclient(hunting): _renew_subs 2024-03-25T19:26:21.515 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.515+0000 7fbc09520640 10 monclient(hunting): authenticate will time out at 6438.787034s 2024-03-25T19:26:21.525 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.525+0000 7fbc037fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fbc0413c950 0x7fbc0414a950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:52454/0 (socket says 172.21.15.83:52454) 2024-03-25T19:26:21.525 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.525+0000 7fbc037fe640 1 -- 172.21.15.83:0/236189283 learned_addr learned my addr 172.21.15.83:0/236189283 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:26:21.541 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.541+0000 7fbc01ffb640 1 -- 172.21.15.83:0/236189283 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1644984072 0 0) 0x7fbc04071a60 con 0x7fbc0413c950 2024-03-25T19:26:21.541 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.541+0000 7fbc01ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:26:21.541 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.541+0000 7fbc01ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:21.541 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.541+0000 7fbc01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:21.541 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.541+0000 7fbc01ffb640 10 monclient(hunting): my global_id is 4518 2024-03-25T19:26:21.541 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.541+0000 7fbc01ffb640 1 -- 172.21.15.83:0/236189283 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbec003520 con 0x7fbc0413c950 2024-03-25T19:26:21.544 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.544+0000 7fbc01ffb640 1 -- 172.21.15.83:0/236189283 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2988798430 0 0) 0x7fbc04071cd0 con 0x7fbc04061830 2024-03-25T19:26:21.544 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.544+0000 7fbc01ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:26:21.544 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.544+0000 7fbc01ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:21.544 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.544+0000 7fbc01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:21.544 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.544+0000 7fbc01ffb640 10 monclient(hunting): my global_id is 4535 2024-03-25T19:26:21.544 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.544+0000 7fbc01ffb640 1 -- 172.21.15.83:0/236189283 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc04071a60 con 0x7fbc04061830 2024-03-25T19:26:21.557 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.557+0000 7fbc01ffb640 1 -- 172.21.15.83:0/236189283 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 499506901 0 0) 0x7fbc04071a60 con 0x7fbc04061830 2024-03-25T19:26:21.557 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.557+0000 7fbc01ffb640 1 -- 172.21.15.83:0/236189283 >> v1:172.21.15.103:6789/0 conn(0x7fbc0406ff10 legacy=0x7fbc040723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:21.557 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.557+0000 7fbc01ffb640 1 -- 172.21.15.83:0/236189283 >> v1:172.21.15.83:6789/0 conn(0x7fbc0413c950 legacy=0x7fbc0414a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:21.557 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.557+0000 7fbc01ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:26:21.557 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.557+0000 7fbc01ffb640 1 monclient: found mon.noname-b 2024-03-25T19:26:21.557 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.557+0000 7fbc01ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:26:21.557 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.557+0000 7fbc01ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.83:6790/0 2024-03-25T19:26:21.557 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.557+0000 7fbc01ffb640 1 -- 172.21.15.83:0/236189283 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc04070690 con 0x7fbc04061830 2024-03-25T19:26:21.557 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.557+0000 7fbc01ffb640 10 monclient: _finish_auth 0 2024-03-25T19:26:21.557 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.557+0000 7fbc01ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:26:21.557 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.557+0000 7fbc01ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:26:21.557 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.557+0000 7fbc01ffb640 1 -- 172.21.15.83:0/236189283 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fbbf4001670 con 0x7fbc04061830 2024-03-25T19:26:21.557 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.557+0000 7fbc09520640 5 monclient: authenticate success, global_id 4535 2024-03-25T19:26:21.558 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.557+0000 7fbc09520640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:26:21.558 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.557+0000 7fbc01ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:21.558 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.557+0000 7fbc01ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:26:21.558 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.557+0000 7fbc01ffb640 10 monclient: dump: 2024-03-25T19:26:21.558 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:26:21.558 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:26:21.558 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:21.558 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:21.558 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:21.558 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:26:21.558 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:26:21.558 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:26:21.558 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:26:21.559 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:21.559 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.557+0000 7fbc09520640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:26:21.573 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.573+0000 7fbc01ffb640 1 -- 172.21.15.83:0/236189283 <== mon.2 v1:172.21.15.83:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbbf40017d0 con 0x7fbc04061830 2024-03-25T19:26:21.573 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.573+0000 7fbc01ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:26:21.573 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.573+0000 7fbc01ffb640 1 -- 172.21.15.83:0/236189283 <== mon.2 v1:172.21.15.83:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fbbf4001ba0 con 0x7fbc04061830 2024-03-25T19:26:21.574 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.573+0000 7fbc09520640 10 monclient: get_monmap_and_config success 2024-03-25T19:26:21.574 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.573+0000 7fbc09520640 4 set_mon_vals no callback set 2024-03-25T19:26:21.574 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.573+0000 7fbc01ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:21.574 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.573+0000 7fbc01ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:26:21.574 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.573+0000 7fbc01ffb640 10 monclient: dump: 2024-03-25T19:26:21.574 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:26:21.574 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:26:21.574 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:21.574 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:21.574 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:21.574 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:26:21.574 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:26:21.575 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:26:21.575 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:26:21.575 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:21.577 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.574+0000 7fbc09520640 10 monclient: shutdown 2024-03-25T19:26:21.577 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.574+0000 7fbc09520640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:26:21.577 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.574+0000 7fbc09520640 1 -- 172.21.15.83:0/236189283 >> v1:172.21.15.83:6790/0 conn(0x7fbc04061830 legacy=0x7fbc0413c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:21.578 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.574+0000 7fbc09520640 1 -- 172.21.15.83:0/236189283 shutdown_connections 2024-03-25T19:26:21.578 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.574+0000 7fbc09520640 1 -- 172.21.15.83:0/236189283 >> 172.21.15.83:0/236189283 conn(0x7fbc040b11e0 msgr2=0x7fbc040b15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:26:21.578 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.574+0000 7fbc09520640 1 -- 172.21.15.83:0/236189283 shutdown_connections 2024-03-25T19:26:21.578 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.574+0000 7fbc09520640 1 -- 172.21.15.83:0/236189283 wait complete. 2024-03-25T19:26:21.578 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.575+0000 7fbc09520640 10 monclient: build_initial_monmap 2024-03-25T19:26:21.578 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.575+0000 7fbc09520640 10 monclient: monmap: 2024-03-25T19:26:21.579 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:26:21.579 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:26:21.579 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:26:21.579 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:26:21.579 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:26:21.579 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:26:21.579 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:26:21.579 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:26:21.579 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:26:21.580 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:21.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.575+0000 7fbc09520640 1 librados: starting msgr at 2024-03-25T19:26:21.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.575+0000 7fbc09520640 1 librados: starting objecter 2024-03-25T19:26:21.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.576+0000 7fbc09520640 1 Processor -- start 2024-03-25T19:26:21.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.576+0000 7fbc09520640 1 -- start start 2024-03-25T19:26:21.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.576+0000 7fbc09520640 1 librados: setting wanted keys 2024-03-25T19:26:21.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.576+0000 7fbc09520640 1 librados: calling monclient init 2024-03-25T19:26:21.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.576+0000 7fbc09520640 10 monclient: init 2024-03-25T19:26:21.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.576+0000 7fbc09520640 10 monclient: _reopen_session rank -1 2024-03-25T19:26:21.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.576+0000 7fbc09520640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:26:21.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.576+0000 7fbc09520640 10 monclient(hunting): picked mon.noname-a con 0x7fbc04061830 addr v1:172.21.15.83:6789/0 2024-03-25T19:26:21.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.576+0000 7fbc09520640 10 monclient(hunting): picked mon.noname-b con 0x7fbc0406ff10 addr v1:172.21.15.103:6789/0 2024-03-25T19:26:21.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.576+0000 7fbc09520640 10 monclient(hunting): picked mon.noname-c con 0x7fbc0413c950 addr v1:172.21.15.83:6790/0 2024-03-25T19:26:21.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.576+0000 7fbc09520640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc041c96f0 con 0x7fbc04061830 2024-03-25T19:26:21.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.576+0000 7fbc09520640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc041c8510 con 0x7fbc0406ff10 2024-03-25T19:26:21.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.576+0000 7fbc09520640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc041c86a0 con 0x7fbc0413c950 2024-03-25T19:26:21.581 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.576+0000 7fbc09520640 10 monclient(hunting): _renew_subs 2024-03-25T19:26:21.581 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.576+0000 7fbc09520640 10 monclient(hunting): authenticate will time out at 6438.848680s 2024-03-25T19:26:21.581 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.582+0000 7fbc02ffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fbc04061830 0x7fbc041c8e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:52488/0 (socket says 172.21.15.83:52488) 2024-03-25T19:26:21.581 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.582+0000 7fbc02ffd640 1 -- 172.21.15.83:0/754904296 learned_addr learned my addr 172.21.15.83:0/754904296 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:26:21.583 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.583+0000 7fbbeb7fe640 10 client.?.objecter ms_handle_connect 0x7fbc04061830 2024-03-25T19:26:21.583 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.583+0000 7fbbeb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:21.589 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.589+0000 7fbbeb7fe640 1 -- 172.21.15.83:0/754904296 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1327058347 0 0) 0x7fbc041c96f0 con 0x7fbc04061830 2024-03-25T19:26:21.589 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.589+0000 7fbbeb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:26:21.589 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.589+0000 7fbbeb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:21.589 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.589+0000 7fbbeb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:21.589 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.589+0000 7fbbeb7fe640 10 monclient(hunting): my global_id is 4521 2024-03-25T19:26:21.589 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.589+0000 7fbbeb7fe640 1 -- 172.21.15.83:0/754904296 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbf0004eb0 con 0x7fbc04061830 2024-03-25T19:26:21.593 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.593+0000 7fbbeb7fe640 10 client.?.objecter ms_handle_connect 0x7fbc0406ff10 2024-03-25T19:26:21.593 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.593+0000 7fbbeb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:21.603 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.603+0000 7fbbeb7fe640 1 -- 172.21.15.83:0/754904296 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1586879767 0 0) 0x7fbbf0004eb0 con 0x7fbc04061830 2024-03-25T19:26:21.603 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.603+0000 7fbbeb7fe640 1 -- 172.21.15.83:0/754904296 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc041c96f0 con 0x7fbc04061830 2024-03-25T19:26:21.603 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.603+0000 7fbbeb7fe640 1 -- 172.21.15.83:0/754904296 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fbbf4001be0 con 0x7fbc04061830 2024-03-25T19:26:21.603 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.603+0000 7fbbeb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:21.603 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.603+0000 7fbbeb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:26:21.603 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.603+0000 7fbbeb7fe640 10 monclient(hunting): dump: 2024-03-25T19:26:21.603 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:26:21.603 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:26:21.604 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:21.604 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:21.604 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:21.604 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:26:21.604 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:26:21.604 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:26:21.604 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:26:21.604 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:21.604 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.603+0000 7fbbeb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:26:21.605 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.605+0000 7fbbeb7fe640 1 -- 172.21.15.83:0/754904296 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 570344365 0 0) 0x7fbc041c8510 con 0x7fbc0406ff10 2024-03-25T19:26:21.605 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.605+0000 7fbbeb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:26:21.605 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.605+0000 7fbbeb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:21.605 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.605+0000 7fbbeb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:21.605 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.605+0000 7fbbeb7fe640 10 monclient(hunting): my global_id is 4531 2024-03-25T19:26:21.605 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.605+0000 7fbbeb7fe640 1 -- 172.21.15.83:0/754904296 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbf0004eb0 con 0x7fbc0406ff10 2024-03-25T19:26:21.610 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.610+0000 7fbbeb7fe640 10 client.?.objecter ms_handle_connect 0x7fbc0413c950 2024-03-25T19:26:21.610 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.610+0000 7fbbeb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:21.619 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.619+0000 7fbbeb7fe640 1 -- 172.21.15.83:0/754904296 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 70333306 0 0) 0x7fbc041c96f0 con 0x7fbc04061830 2024-03-25T19:26:21.619 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.619+0000 7fbbeb7fe640 1 -- 172.21.15.83:0/754904296 >> v1:172.21.15.83:6790/0 conn(0x7fbc0413c950 legacy=0x7fbc041dad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:21.619 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.619+0000 7fbbeb7fe640 1 -- 172.21.15.83:0/754904296 >> v1:172.21.15.103:6789/0 conn(0x7fbc0406ff10 legacy=0x7fbc041da680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:21.620 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.619+0000 7fbbeb7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:26:21.620 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.619+0000 7fbbeb7fe640 1 monclient: found mon.a 2024-03-25T19:26:21.620 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.619+0000 7fbbeb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:26:21.620 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.619+0000 7fbbeb7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:26:21.620 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.619+0000 7fbbeb7fe640 1 -- 172.21.15.83:0/754904296 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc040ada60 con 0x7fbc04061830 2024-03-25T19:26:21.620 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.619+0000 7fbbeb7fe640 10 monclient: _finish_auth 0 2024-03-25T19:26:21.620 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.619+0000 7fbbeb7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:26:21.620 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.619+0000 7fbbeb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:26:21.620 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.619+0000 7fbc09520640 5 monclient: authenticate success, global_id 4521 2024-03-25T19:26:21.620 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.619+0000 7fbc09520640 10 monclient: _renew_subs 2024-03-25T19:26:21.620 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.619+0000 7fbc09520640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:26:21.621 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.619+0000 7fbc09520640 1 -- 172.21.15.83:0/754904296 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc040adcb0 con 0x7fbc04061830 2024-03-25T19:26:21.621 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.619+0000 7fbc09520640 10 client.4521.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:26:21.621 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.619+0000 7fbc09520640 10 monclient: _renew_subs 2024-03-25T19:26:21.621 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.619+0000 7fbc09520640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:26:21.621 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.619+0000 7fbc09520640 1 -- 172.21.15.83:0/754904296 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc041e2df0 con 0x7fbc04061830 2024-03-25T19:26:21.621 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.619+0000 7fbc09520640 1 librados: init done 2024-03-25T19:26:21.622 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.621+0000 7fbc09520640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:21.622 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.621+0000 7fbc09520640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:21.622 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.621+0000 7fbc09520640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:26:21.622 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.621+0000 7fbc09520640 1 -- 172.21.15.83:0/754904296 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc040627b0 con 0x7fbc04061830 2024-03-25T19:26:21.917 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.917+0000 7fbbeb7fe640 1 -- 172.21.15.83:0/754904296 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbbf40038a0 con 0x7fbc04061830 2024-03-25T19:26:21.917 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.917+0000 7fbbeb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:26:21.917 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.917+0000 7fbbeb7fe640 1 -- 172.21.15.83:0/754904296 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fbbf4003c70 con 0x7fbc04061830 2024-03-25T19:26:21.917 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.917+0000 7fbbeb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:21.917 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.917+0000 7fbbeb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:26:21.917 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.917+0000 7fbbeb7fe640 10 monclient: dump: 2024-03-25T19:26:21.918 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:26:21.918 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:26:21.918 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:21.918 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:21.918 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:21.918 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:26:21.918 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:26:21.918 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:26:21.918 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:26:21.918 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:21.918 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.917+0000 7fbc017fa640 4 set_mon_vals no callback set 2024-03-25T19:26:21.919 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.918+0000 7fbbeb7fe640 1 -- 172.21.15.83:0/754904296 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7fbbf4003e90 con 0x7fbc04061830 2024-03-25T19:26:21.919 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.919+0000 7fbc027fc640 10 monclient: get_auth_request con 0x7fbbf006ddd0 auth_method 0 2024-03-25T19:26:21.919 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.919+0000 7fbbeb7fe640 10 client.4521.objecter ms_handle_connect 0x7fbbf006ddd0 2024-03-25T19:26:21.923 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.923+0000 7fbbeb7fe640 1 -- 172.21.15.83:0/754904296 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7fbbf40549c0 con 0x7fbc04061830 2024-03-25T19:26:21.923 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.923+0000 7fbbeb7fe640 10 client.4521.objecter ms_dispatch 0x7fbc04000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:26:21.923 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.923+0000 7fbbeb7fe640 3 client.4521.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:26:21.923 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.923+0000 7fbbeb7fe640 3 client.4521.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:26:21.923 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.923+0000 7fbbeb7fe640 20 client.4521.objecter dump_active .. 0 homeless 2024-03-25T19:26:21.923 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.923+0000 7fbbeb7fe640 1 -- 172.21.15.83:0/754904296 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbbf40504d0 con 0x7fbc04061830 2024-03-25T19:26:21.923 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.923+0000 7fbbeb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:21.924 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:21.923+0000 7fbbeb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:26:22.177 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:22.178+0000 7fbc09520640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:22.178 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:22.178+0000 7fbc09520640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:22.178 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:22.178+0000 7fbc09520640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:26:22.178 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:22.178+0000 7fbc09520640 1 -- 172.21.15.83:0/754904296 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbc04069f40 con 0x7fbc04061830 2024-03-25T19:26:22.210 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:22.211+0000 7fbbeb7fe640 1 -- 172.21.15.83:0/754904296 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7fbbf4052400 con 0x7fbc04061830 2024-03-25T19:26:22.211 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:22.211+0000 7fbbeb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:22.211 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:22.211+0000 7fbbeb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:26:22.211 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:26:22.211 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:26:22.213 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:22.213+0000 7fbc09520640 10 librados: watch_flush enter 2024-03-25T19:26:22.213 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:22.213+0000 7fbc09520640 10 librados: watch_flush exit 2024-03-25T19:26:22.214 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:22.213+0000 7fbc09520640 20 client.4521.objecter shutdown clearing up homeless session... 2024-03-25T19:26:22.214 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:22.213+0000 7fbc09520640 10 client.4521.objecter successfully canceled tick 2024-03-25T19:26:22.214 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:22.213+0000 7fbc09520640 1 -- 172.21.15.83:0/754904296 >> v1:172.21.15.103:6800/345072327 conn(0x7fbbf006ddd0 legacy=0x7fbbf0070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:22.214 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:22.213+0000 7fbc09520640 10 monclient: shutdown 2024-03-25T19:26:22.214 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:22.213+0000 7fbc09520640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:26:22.214 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:22.213+0000 7fbc09520640 1 -- 172.21.15.83:0/754904296 >> v1:172.21.15.83:6789/0 conn(0x7fbc04061830 legacy=0x7fbc041c8e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:22.214 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:22.213+0000 7fbc09520640 1 -- 172.21.15.83:0/754904296 shutdown_connections 2024-03-25T19:26:22.214 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:22.213+0000 7fbc09520640 1 -- 172.21.15.83:0/754904296 >> 172.21.15.83:0/754904296 conn(0x7fbc040b11e0 msgr2=0x7fbc04074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:26:22.214 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:22.213+0000 7fbc09520640 1 -- 172.21.15.83:0/754904296 shutdown_connections 2024-03-25T19:26:22.214 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:22.213+0000 7fbc09520640 1 -- 172.21.15.83:0/754904296 wait complete. 2024-03-25T19:26:22.214 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:22.213+0000 7fbc09520640 1 librados: shutdown 2024-03-25T19:26:22.224 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:26:28.341 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:07:10.268 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:28.394 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:07:10.322 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:28.421 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:07:10.349 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:28.447 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:07:10.374 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:28.463 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:07:10.390 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:28.463 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:07:10.373 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:28.471 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:07:10.381 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:28.490 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:07:10.417 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:28.501 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:07:10.411 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:29.030 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:26:29.142 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.142+0000 7fce12c93640 10 monclient: get_monmap_and_config 2024-03-25T19:26:29.143 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.142+0000 7fce12c93640 10 monclient: build_initial_monmap 2024-03-25T19:26:29.143 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.142+0000 7fce12c93640 10 monclient: monmap: 2024-03-25T19:26:29.143 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:26:29.143 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:26:29.143 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:26:29.143752+0000 2024-03-25T19:26:29.143 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:26:29.143752+0000 2024-03-25T19:26:29.143 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:26:29.143 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:26:29.143 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:26:29.143 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:26:29.143 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:26:29.143 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:29.144 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.143+0000 7fce12c93640 1 Processor -- start 2024-03-25T19:26:29.144 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.144+0000 7fce12c93640 1 -- start start 2024-03-25T19:26:29.144 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.144+0000 7fce12c93640 10 monclient: init 2024-03-25T19:26:29.144 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.144+0000 7fce12c93640 10 monclient: _reopen_session rank -1 2024-03-25T19:26:29.145 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.144+0000 7fce12c93640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:26:29.145 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.144+0000 7fce12c93640 10 monclient(hunting): picked mon.noname-b con 0x7fce0c0a98f0 addr v1:172.21.15.83:6790/0 2024-03-25T19:26:29.145 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.144+0000 7fce12c93640 10 monclient(hunting): picked mon.noname-c con 0x7fce0c13f710 addr v1:172.21.15.103:6789/0 2024-03-25T19:26:29.145 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.144+0000 7fce12c93640 10 monclient(hunting): picked mon.noname-a con 0x7fce0c1430c0 addr v1:172.21.15.83:6789/0 2024-03-25T19:26:29.145 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.144+0000 7fce12c93640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce0c142520 con 0x7fce0c1430c0 2024-03-25T19:26:29.145 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.144+0000 7fce12c93640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce0c0b28a0 con 0x7fce0c13f710 2024-03-25T19:26:29.145 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.144+0000 7fce12c93640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce0c0b2a30 con 0x7fce0c0a98f0 2024-03-25T19:26:29.145 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.144+0000 7fce12c93640 10 monclient(hunting): _renew_subs 2024-03-25T19:26:29.145 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.144+0000 7fce12c93640 10 monclient(hunting): authenticate will time out at 6446.416668s 2024-03-25T19:26:29.151 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.152+0000 7fce10a2f640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fce0c0a98f0 0x7fce0c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:34902/0 (socket says 172.21.15.83:34902) 2024-03-25T19:26:29.151 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.152+0000 7fce10a2f640 1 -- 172.21.15.83:0/3360855573 learned_addr learned my addr 172.21.15.83:0/3360855573 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:26:29.167 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.167+0000 7fce0b7fe640 1 -- 172.21.15.83:0/3360855573 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1354142274 0 0) 0x7fce0c142520 con 0x7fce0c1430c0 2024-03-25T19:26:29.167 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.167+0000 7fce0b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:26:29.167 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.167+0000 7fce0b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:29.167 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.167+0000 7fce0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:29.167 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.167+0000 7fce0b7fe640 10 monclient(hunting): my global_id is 4524 2024-03-25T19:26:29.167 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.167+0000 7fce0b7fe640 1 -- 172.21.15.83:0/3360855573 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdfc003520 con 0x7fce0c1430c0 2024-03-25T19:26:29.179 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.179+0000 7fce0b7fe640 1 -- 172.21.15.83:0/3360855573 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 172412226 0 0) 0x7fce0c0b28a0 con 0x7fce0c13f710 2024-03-25T19:26:29.179 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.179+0000 7fce0b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:26:29.179 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.179+0000 7fce0b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:29.179 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.179+0000 7fce0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:29.179 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.179+0000 7fce0b7fe640 10 monclient(hunting): my global_id is 4534 2024-03-25T19:26:29.179 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.179+0000 7fce0b7fe640 1 -- 172.21.15.83:0/3360855573 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce0c142520 con 0x7fce0c13f710 2024-03-25T19:26:29.179 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.179+0000 7fce0b7fe640 1 -- 172.21.15.83:0/3360855573 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3601181768 0 0) 0x7fce0c0b2a30 con 0x7fce0c0a98f0 2024-03-25T19:26:29.179 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.179+0000 7fce0b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:26:29.179 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.179+0000 7fce0b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:29.179 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.179+0000 7fce0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:29.179 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.179+0000 7fce0b7fe640 10 monclient(hunting): my global_id is 4541 2024-03-25T19:26:29.179 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.179+0000 7fce0b7fe640 1 -- 172.21.15.83:0/3360855573 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce0c0b28a0 con 0x7fce0c0a98f0 2024-03-25T19:26:29.183 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.182+0000 7fce0b7fe640 1 -- 172.21.15.83:0/3360855573 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 900921845 0 0) 0x7fcdfc003520 con 0x7fce0c1430c0 2024-03-25T19:26:29.183 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.183+0000 7fce0b7fe640 1 -- 172.21.15.83:0/3360855573 >> v1:172.21.15.83:6790/0 conn(0x7fce0c0a98f0 legacy=0x7fce0c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:29.183 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.183+0000 7fce0b7fe640 1 -- 172.21.15.83:0/3360855573 >> v1:172.21.15.103:6789/0 conn(0x7fce0c13f710 legacy=0x7fce0c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:29.183 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.183+0000 7fce0b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:26:29.183 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.183+0000 7fce0b7fe640 1 monclient: found mon.noname-a 2024-03-25T19:26:29.183 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.183+0000 7fce0b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:26:29.183 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.183+0000 7fce0b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.83:6789/0 2024-03-25T19:26:29.183 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.183+0000 7fce0b7fe640 1 -- 172.21.15.83:0/3360855573 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce0c0b44c0 con 0x7fce0c1430c0 2024-03-25T19:26:29.183 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.183+0000 7fce0b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:26:29.183 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.183+0000 7fce0b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:26:29.183 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.183+0000 7fce0b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:26:29.184 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.183+0000 7fce0b7fe640 1 -- 172.21.15.83:0/3360855573 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fce04001a10 con 0x7fce0c1430c0 2024-03-25T19:26:29.184 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.183+0000 7fce0b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:29.184 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.183+0000 7fce0b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:26:29.184 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.183+0000 7fce0b7fe640 10 monclient: dump: 2024-03-25T19:26:29.184 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:26:29.184 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:26:29.184 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:29.184 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:29.184 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:29.184 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:26:29.184 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:26:29.184 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:26:29.184 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:26:29.184 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:29.184 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.183+0000 7fce12c93640 5 monclient: authenticate success, global_id 4524 2024-03-25T19:26:29.185 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.183+0000 7fce12c93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:26:29.199 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.200+0000 7fce0b7fe640 1 -- 172.21.15.83:0/3360855573 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce04001b70 con 0x7fce0c1430c0 2024-03-25T19:26:29.200 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.200+0000 7fce0b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:26:29.200 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.200+0000 7fce0b7fe640 1 -- 172.21.15.83:0/3360855573 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fce04001f80 con 0x7fce0c1430c0 2024-03-25T19:26:29.200 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.200+0000 7fce0b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:29.200 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.200+0000 7fce0b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:26:29.200 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.200+0000 7fce0b7fe640 10 monclient: dump: 2024-03-25T19:26:29.200 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:26:29.200 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:26:29.200 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:29.200 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:29.200 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:29.200 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:26:29.201 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:26:29.201 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:26:29.201 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:26:29.201 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:29.201 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.200+0000 7fce12c93640 10 monclient: get_monmap_and_config success 2024-03-25T19:26:29.201 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.200+0000 7fce12c93640 4 set_mon_vals no callback set 2024-03-25T19:26:29.203 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.201+0000 7fce12c93640 10 monclient: shutdown 2024-03-25T19:26:29.203 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.201+0000 7fce12c93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:26:29.203 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.201+0000 7fce12c93640 1 -- 172.21.15.83:0/3360855573 >> v1:172.21.15.83:6789/0 conn(0x7fce0c1430c0 legacy=0x7fce0c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:29.203 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.201+0000 7fce12c93640 1 -- 172.21.15.83:0/3360855573 shutdown_connections 2024-03-25T19:26:29.203 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.201+0000 7fce12c93640 1 -- 172.21.15.83:0/3360855573 >> 172.21.15.83:0/3360855573 conn(0x7fce0c0b7ea0 msgr2=0x7fce0c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:26:29.203 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.201+0000 7fce12c93640 1 -- 172.21.15.83:0/3360855573 shutdown_connections 2024-03-25T19:26:29.204 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.201+0000 7fce12c93640 1 -- 172.21.15.83:0/3360855573 wait complete. 2024-03-25T19:26:29.204 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.201+0000 7fce12c93640 10 monclient: build_initial_monmap 2024-03-25T19:26:29.204 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.201+0000 7fce12c93640 10 monclient: monmap: 2024-03-25T19:26:29.204 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:26:29.204 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:26:29.204 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:26:29.204 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:26:29.204 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:26:29.204 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:26:29.204 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:26:29.204 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:26:29.204 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:26:29.204 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:29.204 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.202+0000 7fce12c93640 1 librados: starting msgr at 2024-03-25T19:26:29.205 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.202+0000 7fce12c93640 1 librados: starting objecter 2024-03-25T19:26:29.205 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.202+0000 7fce12c93640 1 Processor -- start 2024-03-25T19:26:29.205 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.202+0000 7fce12c93640 1 -- start start 2024-03-25T19:26:29.205 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.202+0000 7fce12c93640 1 librados: setting wanted keys 2024-03-25T19:26:29.205 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.202+0000 7fce12c93640 1 librados: calling monclient init 2024-03-25T19:26:29.205 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.202+0000 7fce12c93640 10 monclient: init 2024-03-25T19:26:29.205 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.202+0000 7fce12c93640 10 monclient: _reopen_session rank -1 2024-03-25T19:26:29.205 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.202+0000 7fce12c93640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:26:29.205 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.202+0000 7fce12c93640 10 monclient(hunting): picked mon.noname-b con 0x7fce0c0a98f0 addr v1:172.21.15.103:6789/0 2024-03-25T19:26:29.205 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.202+0000 7fce12c93640 10 monclient(hunting): picked mon.noname-a con 0x7fce0c13f710 addr v1:172.21.15.83:6789/0 2024-03-25T19:26:29.205 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.203+0000 7fce12c93640 10 monclient(hunting): picked mon.noname-c con 0x7fce0c1430c0 addr v1:172.21.15.83:6790/0 2024-03-25T19:26:29.205 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.203+0000 7fce12c93640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce0c13b410 con 0x7fce0c13f710 2024-03-25T19:26:29.205 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.203+0000 7fce12c93640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce0c13b010 con 0x7fce0c0a98f0 2024-03-25T19:26:29.205 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.203+0000 7fce12c93640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce0c135230 con 0x7fce0c1430c0 2024-03-25T19:26:29.205 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.203+0000 7fce12c93640 10 monclient(hunting): _renew_subs 2024-03-25T19:26:29.206 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.203+0000 7fce12c93640 10 monclient(hunting): authenticate will time out at 6446.474821s 2024-03-25T19:26:29.213 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.214+0000 7fce11230640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fce0c1430c0 0x7fce0c1db2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:34950/0 (socket says 172.21.15.83:34950) 2024-03-25T19:26:29.213 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.214+0000 7fce11230640 1 -- 172.21.15.83:0/1141569658 learned_addr learned my addr 172.21.15.83:0/1141569658 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:26:29.215 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.215+0000 7fce097fa640 10 client.?.objecter ms_handle_connect 0x7fce0c1430c0 2024-03-25T19:26:29.215 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.215+0000 7fce097fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:29.219 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.219+0000 7fce097fa640 10 client.?.objecter ms_handle_connect 0x7fce0c0a98f0 2024-03-25T19:26:29.219 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.219+0000 7fce097fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:29.220 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.220+0000 7fce097fa640 10 client.?.objecter ms_handle_connect 0x7fce0c13f710 2024-03-25T19:26:29.220 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.220+0000 7fce097fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:29.226 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.226+0000 7fce097fa640 1 -- 172.21.15.83:0/1141569658 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 884813206 0 0) 0x7fce0c13b410 con 0x7fce0c13f710 2024-03-25T19:26:29.226 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.226+0000 7fce097fa640 10 monclient(hunting): _negotiate 2024-03-25T19:26:29.226 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.226+0000 7fce097fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:29.226 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.226+0000 7fce097fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:29.226 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.226+0000 7fce097fa640 10 monclient(hunting): my global_id is 4527 2024-03-25T19:26:29.226 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.226+0000 7fce097fa640 1 -- 172.21.15.83:0/1141569658 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdf0004eb0 con 0x7fce0c13f710 2024-03-25T19:26:29.228 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.228+0000 7fce097fa640 1 -- 172.21.15.83:0/1141569658 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 388609025 0 0) 0x7fce0c135230 con 0x7fce0c1430c0 2024-03-25T19:26:29.228 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.228+0000 7fce097fa640 10 monclient(hunting): _negotiate 2024-03-25T19:26:29.228 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.228+0000 7fce097fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:29.228 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.228+0000 7fce097fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:29.228 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.228+0000 7fce097fa640 10 monclient(hunting): my global_id is 4544 2024-03-25T19:26:29.228 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.228+0000 7fce097fa640 1 -- 172.21.15.83:0/1141569658 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce0c13b410 con 0x7fce0c1430c0 2024-03-25T19:26:29.229 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.229+0000 7fce097fa640 1 -- 172.21.15.83:0/1141569658 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2792017989 0 0) 0x7fce0c13b010 con 0x7fce0c0a98f0 2024-03-25T19:26:29.229 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.229+0000 7fce097fa640 10 monclient(hunting): _negotiate 2024-03-25T19:26:29.229 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.229+0000 7fce097fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:29.229 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.229+0000 7fce097fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:29.229 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.229+0000 7fce097fa640 10 monclient(hunting): my global_id is 4537 2024-03-25T19:26:29.229 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.229+0000 7fce097fa640 1 -- 172.21.15.83:0/1141569658 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce0c135230 con 0x7fce0c0a98f0 2024-03-25T19:26:29.238 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.238+0000 7fce097fa640 1 -- 172.21.15.83:0/1141569658 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1103600081 0 0) 0x7fcdf0004eb0 con 0x7fce0c13f710 2024-03-25T19:26:29.238 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.238+0000 7fce097fa640 1 -- 172.21.15.83:0/1141569658 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce0c13b010 con 0x7fce0c13f710 2024-03-25T19:26:29.238 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.238+0000 7fce097fa640 1 -- 172.21.15.83:0/1141569658 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fcdf8001970 con 0x7fce0c13f710 2024-03-25T19:26:29.238 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.238+0000 7fce097fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:29.238 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.238+0000 7fce097fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:26:29.238 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.238+0000 7fce097fa640 10 monclient(hunting): dump: 2024-03-25T19:26:29.239 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:26:29.239 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:26:29.239 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:29.239 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:29.239 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:29.239 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:26:29.239 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:26:29.239 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:26:29.239 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:26:29.239 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:29.239 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.238+0000 7fce097fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:26:29.256 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.256+0000 7fce097fa640 1 -- 172.21.15.83:0/1141569658 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 473993370 0 0) 0x7fce0c13b410 con 0x7fce0c1430c0 2024-03-25T19:26:29.256 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.256+0000 7fce097fa640 1 -- 172.21.15.83:0/1141569658 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcdf0004eb0 con 0x7fce0c1430c0 2024-03-25T19:26:29.256 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.256+0000 7fce097fa640 1 -- 172.21.15.83:0/1141569658 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fce04001e80 con 0x7fce0c1430c0 2024-03-25T19:26:29.256 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.256+0000 7fce097fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:29.256 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.256+0000 7fce097fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:26:29.256 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.256+0000 7fce097fa640 10 monclient(hunting): dump: 2024-03-25T19:26:29.256 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:26:29.256 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:26:29.256 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:29.256 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:29.257 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:29.257 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:26:29.257 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:26:29.257 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:26:29.257 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:26:29.257 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:29.257 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.258+0000 7fce097fa640 1 -- 172.21.15.83:0/1141569658 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1324873668 0 0) 0x7fce0c135230 con 0x7fce0c0a98f0 2024-03-25T19:26:29.258 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.258+0000 7fce097fa640 1 -- 172.21.15.83:0/1141569658 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce0c13b410 con 0x7fce0c0a98f0 2024-03-25T19:26:29.258 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.258+0000 7fce097fa640 1 -- 172.21.15.83:0/1141569658 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fce00001830 con 0x7fce0c0a98f0 2024-03-25T19:26:29.258 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.258+0000 7fce097fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:29.258 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.258+0000 7fce097fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:26:29.258 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.258+0000 7fce097fa640 10 monclient(hunting): dump: 2024-03-25T19:26:29.258 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:26:29.258 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:26:29.258 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:29.258 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:29.258 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:29.258 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:26:29.258 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:26:29.259 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:26:29.259 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:26:29.259 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:29.266 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.266+0000 7fce097fa640 1 -- 172.21.15.83:0/1141569658 <== mon.1 v1:172.21.15.103:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2513746197 0 0) 0x7fce0c13b410 con 0x7fce0c0a98f0 2024-03-25T19:26:29.266 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.266+0000 7fce097fa640 1 -- 172.21.15.83:0/1141569658 >> v1:172.21.15.83:6790/0 conn(0x7fce0c1430c0 legacy=0x7fce0c1db2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:29.266 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.266+0000 7fce097fa640 1 -- 172.21.15.83:0/1141569658 >> v1:172.21.15.83:6789/0 conn(0x7fce0c13f710 legacy=0x7fce0c1dabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:29.266 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.266+0000 7fce097fa640 10 monclient: _finish_hunting 0 2024-03-25T19:26:29.266 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.266+0000 7fce097fa640 1 monclient: found mon.b 2024-03-25T19:26:29.266 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.266+0000 7fce097fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:26:29.266 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.266+0000 7fce097fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:26:29.266 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.266+0000 7fce097fa640 1 -- 172.21.15.83:0/1141569658 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce0c1353c0 con 0x7fce0c0a98f0 2024-03-25T19:26:29.266 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.266+0000 7fce097fa640 10 monclient: _finish_auth 0 2024-03-25T19:26:29.266 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.266+0000 7fce097fa640 10 monclient: _check_auth_tickets 2024-03-25T19:26:29.266 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.266+0000 7fce097fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:26:29.267 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.266+0000 7fce12c93640 5 monclient: authenticate success, global_id 4537 2024-03-25T19:26:29.267 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.266+0000 7fce12c93640 10 monclient: _renew_subs 2024-03-25T19:26:29.267 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.266+0000 7fce12c93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:26:29.267 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.266+0000 7fce12c93640 1 -- 172.21.15.83:0/1141569658 --> v1:172.21.15.103:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce0c0b78e0 con 0x7fce0c0a98f0 2024-03-25T19:26:29.267 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.266+0000 7fce12c93640 10 client.4537.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:26:29.267 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.266+0000 7fce12c93640 10 monclient: _renew_subs 2024-03-25T19:26:29.267 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.266+0000 7fce12c93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:26:29.267 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.266+0000 7fce12c93640 1 -- 172.21.15.83:0/1141569658 --> v1:172.21.15.103:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fce0c1315f0 con 0x7fce0c0a98f0 2024-03-25T19:26:29.267 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.266+0000 7fce12c93640 1 librados: init done 2024-03-25T19:26:29.268 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.267+0000 7fce12c93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:29.268 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.267+0000 7fce12c93640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:29.268 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.267+0000 7fce12c93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:26:29.268 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.267+0000 7fce12c93640 1 -- 172.21.15.83:0/1141569658 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce0c0627b0 con 0x7fce0c0a98f0 2024-03-25T19:26:29.566 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.566+0000 7fce097fa640 1 -- 172.21.15.83:0/1141569658 <== mon.1 v1:172.21.15.103:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce00001d40 con 0x7fce0c0a98f0 2024-03-25T19:26:29.567 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.566+0000 7fce097fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:26:29.567 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.566+0000 7fce097fa640 1 -- 172.21.15.83:0/1141569658 <== mon.1 v1:172.21.15.103:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fce000049c0 con 0x7fce0c0a98f0 2024-03-25T19:26:29.567 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.566+0000 7fce097fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:29.567 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.566+0000 7fce097fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:26:29.567 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.566+0000 7fce097fa640 10 monclient: dump: 2024-03-25T19:26:29.567 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:26:29.567 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:26:29.567 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:29.567 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:29.567 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:29.567 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:26:29.567 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:26:29.568 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:26:29.568 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:26:29.568 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:29.568 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.566+0000 7fce0affd640 4 set_mon_vals no callback set 2024-03-25T19:26:29.569 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.568+0000 7fce097fa640 1 -- 172.21.15.83:0/1141569658 <== mon.1 v1:172.21.15.103:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7fce0001aa50 con 0x7fce0c0a98f0 2024-03-25T19:26:29.569 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.568+0000 7fce0bfff640 10 monclient: get_auth_request con 0x7fcdf006e160 auth_method 0 2024-03-25T19:26:29.569 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.568+0000 7fce097fa640 10 client.4537.objecter ms_handle_connect 0x7fcdf006e160 2024-03-25T19:26:29.573 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.573+0000 7fce097fa640 1 -- 172.21.15.83:0/1141569658 <== mon.1 v1:172.21.15.103:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7fce00054730 con 0x7fce0c0a98f0 2024-03-25T19:26:29.573 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.573+0000 7fce097fa640 10 client.4537.objecter ms_dispatch 0x7fce0c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:26:29.573 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.573+0000 7fce097fa640 3 client.4537.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:26:29.573 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.573+0000 7fce097fa640 3 client.4537.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:26:29.573 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.573+0000 7fce097fa640 20 client.4537.objecter dump_active .. 0 homeless 2024-03-25T19:26:29.573 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.573+0000 7fce097fa640 1 -- 172.21.15.83:0/1141569658 <== mon.1 v1:172.21.15.103:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fce00050070 con 0x7fce0c0a98f0 2024-03-25T19:26:29.573 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.573+0000 7fce097fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:29.573 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.573+0000 7fce097fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:26:29.799 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.799+0000 7fce12c93640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:29.799 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.799+0000 7fce12c93640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:29.800 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.799+0000 7fce12c93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:26:29.800 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.799+0000 7fce12c93640 1 -- 172.21.15.83:0/1141569658 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fce0c13b1a0 con 0x7fce0c0a98f0 2024-03-25T19:26:29.853 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.853+0000 7fce097fa640 1 -- 172.21.15.83:0/1141569658 <== mon.1 v1:172.21.15.103:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7fce00052170 con 0x7fce0c0a98f0 2024-03-25T19:26:29.853 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.853+0000 7fce097fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:29.853 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.853+0000 7fce097fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:26:29.853 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:26:29.854 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:26:29.856 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.855+0000 7fce12c93640 10 librados: watch_flush enter 2024-03-25T19:26:29.856 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.855+0000 7fce12c93640 10 librados: watch_flush exit 2024-03-25T19:26:29.856 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.855+0000 7fce12c93640 20 client.4537.objecter shutdown clearing up homeless session... 2024-03-25T19:26:29.856 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.855+0000 7fce12c93640 10 client.4537.objecter successfully canceled tick 2024-03-25T19:26:29.856 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.855+0000 7fce12c93640 1 -- 172.21.15.83:0/1141569658 >> v1:172.21.15.103:6800/345072327 conn(0x7fcdf006e160 legacy=0x7fcdf0070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:29.856 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.856+0000 7fce12c93640 10 monclient: shutdown 2024-03-25T19:26:29.856 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.856+0000 7fce12c93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:26:29.856 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.856+0000 7fce12c93640 1 -- 172.21.15.83:0/1141569658 >> v1:172.21.15.103:6789/0 conn(0x7fce0c0a98f0 legacy=0x7fce0c130ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:29.856 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.856+0000 7fce12c93640 1 -- 172.21.15.83:0/1141569658 shutdown_connections 2024-03-25T19:26:29.856 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.856+0000 7fce12c93640 1 -- 172.21.15.83:0/1141569658 >> 172.21.15.83:0/1141569658 conn(0x7fce0c0b7ea0 msgr2=0x7fce0c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:26:29.857 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.856+0000 7fce12c93640 1 -- 172.21.15.83:0/1141569658 shutdown_connections 2024-03-25T19:26:29.857 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.856+0000 7fce12c93640 1 -- 172.21.15.83:0/1141569658 wait complete. 2024-03-25T19:26:29.857 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:29.856+0000 7fce12c93640 1 librados: shutdown 2024-03-25T19:26:29.866 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:26:36.673 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:26:36.788 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.788+0000 7f24bf496640 10 monclient: get_monmap_and_config 2024-03-25T19:26:36.788 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.788+0000 7f24bf496640 10 monclient: build_initial_monmap 2024-03-25T19:26:36.788 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.788+0000 7f24bf496640 10 monclient: monmap: 2024-03-25T19:26:36.788 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:26:36.788 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:26:36.788 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:26:36.789453+0000 2024-03-25T19:26:36.788 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:26:36.789453+0000 2024-03-25T19:26:36.788 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:26:36.788 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:26:36.788 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:26:36.788 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:26:36.789 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:26:36.789 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:36.789 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.789+0000 7f24bf496640 1 Processor -- start 2024-03-25T19:26:36.790 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.789+0000 7f24bf496640 1 -- start start 2024-03-25T19:26:36.790 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.789+0000 7f24bf496640 10 monclient: init 2024-03-25T19:26:36.790 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.790+0000 7f24bf496640 10 monclient: _reopen_session rank -1 2024-03-25T19:26:36.790 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.790+0000 7f24bf496640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:26:36.790 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.790+0000 7f24bf496640 10 monclient(hunting): picked mon.noname-b con 0x7f24b80a98f0 addr v1:172.21.15.83:6790/0 2024-03-25T19:26:36.790 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.790+0000 7f24bf496640 10 monclient(hunting): picked mon.noname-c con 0x7f24b813f710 addr v1:172.21.15.103:6789/0 2024-03-25T19:26:36.790 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.790+0000 7f24bf496640 10 monclient(hunting): picked mon.noname-a con 0x7f24b81430c0 addr v1:172.21.15.83:6789/0 2024-03-25T19:26:36.790 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.790+0000 7f24bf496640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24b8142520 con 0x7f24b81430c0 2024-03-25T19:26:36.790 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.790+0000 7f24bf496640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24b80b28a0 con 0x7f24b813f710 2024-03-25T19:26:36.790 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.790+0000 7f24bf496640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24b80b2a30 con 0x7f24b80a98f0 2024-03-25T19:26:36.790 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.790+0000 7f24bf496640 10 monclient(hunting): _renew_subs 2024-03-25T19:26:36.790 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.790+0000 7f24bf496640 10 monclient(hunting): authenticate will time out at 6454.062007s 2024-03-25T19:26:36.798 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.798+0000 7f24bda33640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f24b81430c0 0x7f24b814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:45378/0 (socket says 172.21.15.83:45378) 2024-03-25T19:26:36.798 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.798+0000 7f24bda33640 1 -- 172.21.15.83:0/2149571557 learned_addr learned my addr 172.21.15.83:0/2149571557 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:26:36.812 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.812+0000 7f24affff640 1 -- 172.21.15.83:0/2149571557 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 648409443 0 0) 0x7f24b80b28a0 con 0x7f24b813f710 2024-03-25T19:26:36.812 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.813+0000 7f24affff640 10 monclient(hunting): _negotiate 2024-03-25T19:26:36.812 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.813+0000 7f24affff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:36.812 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.813+0000 7f24affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:36.812 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.813+0000 7f24affff640 10 monclient(hunting): my global_id is 4540 2024-03-25T19:26:36.812 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.813+0000 7f24affff640 1 -- 172.21.15.83:0/2149571557 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24a4003520 con 0x7f24b813f710 2024-03-25T19:26:36.818 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.818+0000 7f24affff640 1 -- 172.21.15.83:0/2149571557 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 760668160 0 0) 0x7f24b8142520 con 0x7f24b81430c0 2024-03-25T19:26:36.818 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.818+0000 7f24affff640 10 monclient(hunting): _negotiate 2024-03-25T19:26:36.818 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.818+0000 7f24affff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:36.818 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.818+0000 7f24affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:36.818 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.818+0000 7f24affff640 10 monclient(hunting): my global_id is 4530 2024-03-25T19:26:36.818 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.818+0000 7f24affff640 1 -- 172.21.15.83:0/2149571557 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24b80b28a0 con 0x7f24b81430c0 2024-03-25T19:26:36.822 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.822+0000 7f24affff640 1 -- 172.21.15.83:0/2149571557 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2314792410 0 0) 0x7f24a4003520 con 0x7f24b813f710 2024-03-25T19:26:36.822 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.822+0000 7f24affff640 1 -- 172.21.15.83:0/2149571557 >> v1:172.21.15.83:6790/0 conn(0x7f24b80a98f0 legacy=0x7f24b80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:36.822 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.822+0000 7f24affff640 1 -- 172.21.15.83:0/2149571557 >> v1:172.21.15.83:6789/0 conn(0x7f24b81430c0 legacy=0x7f24b814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:36.822 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.822+0000 7f24affff640 10 monclient: _finish_hunting 0 2024-03-25T19:26:36.822 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.822+0000 7f24affff640 1 monclient: found mon.noname-c 2024-03-25T19:26:36.822 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.822+0000 7f24affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:26:36.822 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.822+0000 7f24affff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.103:6789/0 2024-03-25T19:26:36.822 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.822+0000 7f24affff640 1 -- 172.21.15.83:0/2149571557 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24b80b44c0 con 0x7f24b813f710 2024-03-25T19:26:36.822 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.822+0000 7f24affff640 10 monclient: _finish_auth 0 2024-03-25T19:26:36.822 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.822+0000 7f24affff640 10 monclient: _check_auth_tickets 2024-03-25T19:26:36.822 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.822+0000 7f24affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:26:36.822 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.822+0000 7f24affff640 1 -- 172.21.15.83:0/2149571557 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f24a8001690 con 0x7f24b813f710 2024-03-25T19:26:36.822 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.822+0000 7f24bf496640 5 monclient: authenticate success, global_id 4540 2024-03-25T19:26:36.822 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.822+0000 7f24bf496640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:26:36.823 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.822+0000 7f24affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:36.823 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.822+0000 7f24affff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:26:36.823 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.822+0000 7f24affff640 10 monclient: dump: 2024-03-25T19:26:36.823 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:26:36.823 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:26:36.823 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:36.823 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:36.823 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:36.823 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:26:36.823 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:26:36.823 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:26:36.823 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:26:36.823 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:36.823 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.822+0000 7f24bf496640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:26:36.840 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.840+0000 7f24affff640 1 -- 172.21.15.83:0/2149571557 <== mon.1 v1:172.21.15.103:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24a80017f0 con 0x7f24b813f710 2024-03-25T19:26:36.840 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.840+0000 7f24affff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:26:36.840 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.840+0000 7f24affff640 1 -- 172.21.15.83:0/2149571557 <== mon.1 v1:172.21.15.103:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f24a8001c00 con 0x7f24b813f710 2024-03-25T19:26:36.840 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.840+0000 7f24affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:36.840 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.840+0000 7f24affff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:26:36.840 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.840+0000 7f24affff640 10 monclient: dump: 2024-03-25T19:26:36.840 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:26:36.840 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:26:36.841 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:36.841 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:36.841 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:36.841 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:26:36.841 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:26:36.841 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:26:36.841 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:26:36.841 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:36.841 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.840+0000 7f24bf496640 10 monclient: get_monmap_and_config success 2024-03-25T19:26:36.841 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.840+0000 7f24bf496640 4 set_mon_vals no callback set 2024-03-25T19:26:36.843 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.841+0000 7f24bf496640 10 monclient: shutdown 2024-03-25T19:26:36.843 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.841+0000 7f24bf496640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:26:36.843 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.841+0000 7f24bf496640 1 -- 172.21.15.83:0/2149571557 >> v1:172.21.15.103:6789/0 conn(0x7f24b813f710 legacy=0x7f24b81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:36.843 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.841+0000 7f24bf496640 1 -- 172.21.15.83:0/2149571557 shutdown_connections 2024-03-25T19:26:36.843 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.841+0000 7f24bf496640 1 -- 172.21.15.83:0/2149571557 >> 172.21.15.83:0/2149571557 conn(0x7f24b80b7ea0 msgr2=0x7f24b80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:26:36.843 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.841+0000 7f24bf496640 1 -- 172.21.15.83:0/2149571557 shutdown_connections 2024-03-25T19:26:36.843 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.841+0000 7f24bf496640 1 -- 172.21.15.83:0/2149571557 wait complete. 2024-03-25T19:26:36.844 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.842+0000 7f24bf496640 10 monclient: build_initial_monmap 2024-03-25T19:26:36.844 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.842+0000 7f24bf496640 10 monclient: monmap: 2024-03-25T19:26:36.844 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:26:36.844 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:26:36.844 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:26:36.844 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:26:36.844 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:26:36.844 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:26:36.844 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:26:36.844 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:26:36.844 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:26:36.844 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:36.844 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.842+0000 7f24bf496640 1 librados: starting msgr at 2024-03-25T19:26:36.844 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.842+0000 7f24bf496640 1 librados: starting objecter 2024-03-25T19:26:36.844 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.842+0000 7f24bf496640 1 Processor -- start 2024-03-25T19:26:36.844 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.842+0000 7f24bf496640 1 -- start start 2024-03-25T19:26:36.844 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.842+0000 7f24bf496640 1 librados: setting wanted keys 2024-03-25T19:26:36.845 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.842+0000 7f24bf496640 1 librados: calling monclient init 2024-03-25T19:26:36.845 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.842+0000 7f24bf496640 10 monclient: init 2024-03-25T19:26:36.845 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.843+0000 7f24bf496640 10 monclient: _reopen_session rank -1 2024-03-25T19:26:36.845 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.843+0000 7f24bf496640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:26:36.845 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.843+0000 7f24bf496640 10 monclient(hunting): picked mon.noname-b con 0x7f24b80a98f0 addr v1:172.21.15.103:6789/0 2024-03-25T19:26:36.845 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.843+0000 7f24bf496640 10 monclient(hunting): picked mon.noname-a con 0x7f24b813f710 addr v1:172.21.15.83:6789/0 2024-03-25T19:26:36.845 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.843+0000 7f24bf496640 10 monclient(hunting): picked mon.noname-c con 0x7f24b81430c0 addr v1:172.21.15.83:6790/0 2024-03-25T19:26:36.845 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.843+0000 7f24bf496640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24b81d7490 con 0x7f24b813f710 2024-03-25T19:26:36.845 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.843+0000 7f24bf496640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24b81d76c0 con 0x7f24b80a98f0 2024-03-25T19:26:36.845 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.843+0000 7f24bf496640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24b81d82f0 con 0x7f24b81430c0 2024-03-25T19:26:36.845 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.843+0000 7f24bf496640 10 monclient(hunting): _renew_subs 2024-03-25T19:26:36.845 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.843+0000 7f24bf496640 10 monclient(hunting): authenticate will time out at 6454.115440s 2024-03-25T19:26:36.849 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.850+0000 7f24bca31640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f24b813f710 0x7f24b81383e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:45426/0 (socket says 172.21.15.83:45426) 2024-03-25T19:26:36.849 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.850+0000 7f24bca31640 1 -- 172.21.15.83:0/3380535256 learned_addr learned my addr 172.21.15.83:0/3380535256 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:26:36.850 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.851+0000 7f24adffb640 10 client.?.objecter ms_handle_connect 0x7f24b813f710 2024-03-25T19:26:36.850 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.851+0000 7f24adffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:36.858 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.859+0000 7f24adffb640 10 client.?.objecter ms_handle_connect 0x7f24b81430c0 2024-03-25T19:26:36.858 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.859+0000 7f24adffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:36.859 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.859+0000 7f24adffb640 10 client.?.objecter ms_handle_connect 0x7f24b80a98f0 2024-03-25T19:26:36.859 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.859+0000 7f24adffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:36.876 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.876+0000 7f24adffb640 1 -- 172.21.15.83:0/3380535256 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 62903537 0 0) 0x7f24b81d7490 con 0x7f24b813f710 2024-03-25T19:26:36.876 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.876+0000 7f24adffb640 10 monclient(hunting): _negotiate 2024-03-25T19:26:36.877 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.876+0000 7f24adffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:36.877 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.876+0000 7f24adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:36.877 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.876+0000 7f24adffb640 10 monclient(hunting): my global_id is 4533 2024-03-25T19:26:36.877 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.877+0000 7f24adffb640 1 -- 172.21.15.83:0/3380535256 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f249c004eb0 con 0x7f24b813f710 2024-03-25T19:26:36.877 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.877+0000 7f24adffb640 1 -- 172.21.15.83:0/3380535256 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3182348346 0 0) 0x7f24b81d82f0 con 0x7f24b81430c0 2024-03-25T19:26:36.877 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.877+0000 7f24adffb640 10 monclient(hunting): _negotiate 2024-03-25T19:26:36.877 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.877+0000 7f24adffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:36.877 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.877+0000 7f24adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:36.877 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.877+0000 7f24adffb640 10 monclient(hunting): my global_id is 4550 2024-03-25T19:26:36.877 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.877+0000 7f24adffb640 1 -- 172.21.15.83:0/3380535256 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24b81d7490 con 0x7f24b81430c0 2024-03-25T19:26:36.878 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.878+0000 7f24adffb640 1 -- 172.21.15.83:0/3380535256 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1729270720 0 0) 0x7f24b81d76c0 con 0x7f24b80a98f0 2024-03-25T19:26:36.878 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.878+0000 7f24adffb640 10 monclient(hunting): _negotiate 2024-03-25T19:26:36.878 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.878+0000 7f24adffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:36.878 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.878+0000 7f24adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:36.878 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.878+0000 7f24adffb640 10 monclient(hunting): my global_id is 4543 2024-03-25T19:26:36.878 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.878+0000 7f24adffb640 1 -- 172.21.15.83:0/3380535256 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24b81d82f0 con 0x7f24b80a98f0 2024-03-25T19:26:36.886 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.887+0000 7f24adffb640 1 -- 172.21.15.83:0/3380535256 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 555746193 0 0) 0x7f24b81d82f0 con 0x7f24b80a98f0 2024-03-25T19:26:36.887 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.887+0000 7f24adffb640 1 -- 172.21.15.83:0/3380535256 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24b81d76c0 con 0x7f24b80a98f0 2024-03-25T19:26:36.887 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.887+0000 7f24adffb640 1 -- 172.21.15.83:0/3380535256 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f24b4001860 con 0x7f24b80a98f0 2024-03-25T19:26:36.887 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.887+0000 7f24adffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:36.887 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.887+0000 7f24adffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:26:36.887 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.887+0000 7f24adffb640 10 monclient(hunting): dump: 2024-03-25T19:26:36.887 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:26:36.887 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:26:36.887 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:36.887 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:36.887 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:36.887 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:26:36.887 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:26:36.887 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:26:36.887 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:26:36.887 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:36.887 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.887+0000 7f24adffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:26:36.889 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.890+0000 7f24adffb640 1 -- 172.21.15.83:0/3380535256 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2305196777 0 0) 0x7f249c004eb0 con 0x7f24b813f710 2024-03-25T19:26:36.889 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.890+0000 7f24adffb640 1 -- 172.21.15.83:0/3380535256 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24b81d82f0 con 0x7f24b813f710 2024-03-25T19:26:36.890 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.890+0000 7f24adffb640 1 -- 172.21.15.83:0/3380535256 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f24a8001690 con 0x7f24b813f710 2024-03-25T19:26:36.890 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.890+0000 7f24adffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:36.890 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.890+0000 7f24adffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:26:36.890 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.890+0000 7f24adffb640 10 monclient(hunting): dump: 2024-03-25T19:26:36.890 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:26:36.890 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:26:36.890 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:36.890 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:36.890 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:36.890 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:26:36.890 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:26:36.890 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:26:36.890 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:26:36.890 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:36.896 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.896+0000 7f24adffb640 1 -- 172.21.15.83:0/3380535256 <== mon.1 v1:172.21.15.103:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1773072355 0 0) 0x7f24b81d76c0 con 0x7f24b80a98f0 2024-03-25T19:26:36.896 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.896+0000 7f24adffb640 1 -- 172.21.15.83:0/3380535256 >> v1:172.21.15.83:6790/0 conn(0x7f24b81430c0 legacy=0x7f24b8133fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:36.896 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.896+0000 7f24adffb640 1 -- 172.21.15.83:0/3380535256 >> v1:172.21.15.83:6789/0 conn(0x7f24b813f710 legacy=0x7f24b81383e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:36.896 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.896+0000 7f24adffb640 10 monclient: _finish_hunting 0 2024-03-25T19:26:36.897 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.896+0000 7f24adffb640 1 monclient: found mon.b 2024-03-25T19:26:36.897 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.896+0000 7f24adffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:26:36.897 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.896+0000 7f24adffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:26:36.897 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.896+0000 7f24adffb640 1 -- 172.21.15.83:0/3380535256 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24b8138af0 con 0x7f24b80a98f0 2024-03-25T19:26:36.897 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.896+0000 7f24adffb640 10 monclient: _finish_auth 0 2024-03-25T19:26:36.897 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.896+0000 7f24adffb640 10 monclient: _check_auth_tickets 2024-03-25T19:26:36.897 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.896+0000 7f24adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:26:36.897 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.896+0000 7f24bf496640 5 monclient: authenticate success, global_id 4543 2024-03-25T19:26:36.897 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.896+0000 7f24bf496640 10 monclient: _renew_subs 2024-03-25T19:26:36.897 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.896+0000 7f24bf496640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:26:36.897 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.896+0000 7f24bf496640 1 -- 172.21.15.83:0/3380535256 --> v1:172.21.15.103:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24b81376d0 con 0x7f24b80a98f0 2024-03-25T19:26:36.897 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.896+0000 7f24bf496640 10 client.4543.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:26:36.897 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.897+0000 7f24bf496640 10 monclient: _renew_subs 2024-03-25T19:26:36.897 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.897+0000 7f24bf496640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:26:36.897 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.897+0000 7f24bf496640 1 -- 172.21.15.83:0/3380535256 --> v1:172.21.15.103:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f24b8134fc0 con 0x7f24b80a98f0 2024-03-25T19:26:36.897 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.897+0000 7f24bf496640 1 librados: init done 2024-03-25T19:26:36.898 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.897+0000 7f24bf496640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:36.898 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.898+0000 7f24bf496640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:36.898 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.898+0000 7f24bf496640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:26:36.898 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:36.898+0000 7f24bf496640 1 -- 172.21.15.83:0/3380535256 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24b81d7850 con 0x7f24b80a98f0 2024-03-25T19:26:37.177 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:37.178+0000 7f24adffb640 1 -- 172.21.15.83:0/3380535256 <== mon.1 v1:172.21.15.103:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24b4001d30 con 0x7f24b80a98f0 2024-03-25T19:26:37.177 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:37.178+0000 7f24adffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:26:37.178 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:37.178+0000 7f24adffb640 1 -- 172.21.15.83:0/3380535256 <== mon.1 v1:172.21.15.103:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f24b4003790 con 0x7f24b80a98f0 2024-03-25T19:26:37.178 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:37.178+0000 7f24adffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:37.178 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:37.178+0000 7f24adffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:26:37.178 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:37.178+0000 7f24adffb640 10 monclient: dump: 2024-03-25T19:26:37.178 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:26:37.178 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:26:37.178 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:37.178 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:37.178 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:37.178 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:26:37.178 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:26:37.178 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:26:37.178 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:26:37.178 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:37.179 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:37.178+0000 7f24af7fe640 4 set_mon_vals no callback set 2024-03-25T19:26:37.179 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:37.179+0000 7f24adffb640 1 -- 172.21.15.83:0/3380535256 <== mon.1 v1:172.21.15.103:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7f24b40197a0 con 0x7f24b80a98f0 2024-03-25T19:26:37.179 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:37.179+0000 7f24bca31640 10 monclient: get_auth_request con 0x7f249c06df20 auth_method 0 2024-03-25T19:26:37.179 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:37.180+0000 7f24adffb640 10 client.4543.objecter ms_handle_connect 0x7f249c06df20 2024-03-25T19:26:37.186 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:37.186+0000 7f24adffb640 1 -- 172.21.15.83:0/3380535256 <== mon.1 v1:172.21.15.103:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7f24b4053510 con 0x7f24b80a98f0 2024-03-25T19:26:37.186 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:37.186+0000 7f24adffb640 10 client.4543.objecter ms_dispatch 0x7f24b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:26:37.186 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:37.186+0000 7f24adffb640 3 client.4543.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:26:37.186 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:37.186+0000 7f24adffb640 3 client.4543.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:26:37.186 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:37.186+0000 7f24adffb640 20 client.4543.objecter dump_active .. 0 homeless 2024-03-25T19:26:37.186 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:37.186+0000 7f24adffb640 1 -- 172.21.15.83:0/3380535256 <== mon.1 v1:172.21.15.103:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f24b404ee50 con 0x7f24b80a98f0 2024-03-25T19:26:37.186 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:37.186+0000 7f24adffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:37.186 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:37.186+0000 7f24adffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:26:37.411 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:37.411+0000 7f24bf496640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:37.411 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:37.411+0000 7f24bf496640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:37.411 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:37.411+0000 7f24bf496640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:26:37.411 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:37.411+0000 7f24bf496640 1 -- 172.21.15.83:0/3380535256 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f24b814f300 con 0x7f24b80a98f0 2024-03-25T19:26:37.550 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:37.550+0000 7f24adffb640 1 -- 172.21.15.83:0/3380535256 <== mon.1 v1:172.21.15.103:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7f24b4050f50 con 0x7f24b80a98f0 2024-03-25T19:26:37.551 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:37.551+0000 7f24adffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:37.551 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:37.551+0000 7f24adffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:26:37.551 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:26:37.551 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:26:37.554 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:37.553+0000 7f24bf496640 10 librados: watch_flush enter 2024-03-25T19:26:37.554 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:37.553+0000 7f24bf496640 10 librados: watch_flush exit 2024-03-25T19:26:37.554 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:37.553+0000 7f24bf496640 20 client.4543.objecter shutdown clearing up homeless session... 2024-03-25T19:26:37.554 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:37.553+0000 7f24bf496640 10 client.4543.objecter successfully canceled tick 2024-03-25T19:26:37.554 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:37.553+0000 7f24bf496640 1 -- 172.21.15.83:0/3380535256 >> v1:172.21.15.103:6800/345072327 conn(0x7f249c06df20 legacy=0x7f249c0703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:37.554 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:37.553+0000 7f24bf496640 10 monclient: shutdown 2024-03-25T19:26:37.554 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:37.553+0000 7f24bf496640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:26:37.554 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:37.553+0000 7f24bf496640 1 -- 172.21.15.83:0/3380535256 >> v1:172.21.15.103:6789/0 conn(0x7f24b80a98f0 legacy=0x7f24b81d7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:37.554 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:37.553+0000 7f24bf496640 1 -- 172.21.15.83:0/3380535256 shutdown_connections 2024-03-25T19:26:37.554 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:37.553+0000 7f24bf496640 1 -- 172.21.15.83:0/3380535256 >> 172.21.15.83:0/3380535256 conn(0x7f24b80b7ea0 msgr2=0x7f24b8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:26:37.554 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:37.554+0000 7f24bf496640 1 -- 172.21.15.83:0/3380535256 shutdown_connections 2024-03-25T19:26:37.554 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:37.554+0000 7f24bf496640 1 -- 172.21.15.83:0/3380535256 wait complete. 2024-03-25T19:26:37.554 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:37.554+0000 7f24bf496640 1 librados: shutdown 2024-03-25T19:26:37.566 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:26:44.372 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:26:44.488 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.489+0000 7f4302e7d640 10 monclient: get_monmap_and_config 2024-03-25T19:26:44.489 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.489+0000 7f4302e7d640 10 monclient: build_initial_monmap 2024-03-25T19:26:44.489 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.489+0000 7f4302e7d640 10 monclient: monmap: 2024-03-25T19:26:44.489 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:26:44.489 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:26:44.489 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:26:44.490197+0000 2024-03-25T19:26:44.489 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:26:44.490197+0000 2024-03-25T19:26:44.489 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:26:44.489 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:26:44.489 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:26:44.489 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:26:44.489 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:26:44.489 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:44.491 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.490+0000 7f4302e7d640 1 Processor -- start 2024-03-25T19:26:44.491 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.490+0000 7f4302e7d640 1 -- start start 2024-03-25T19:26:44.491 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.490+0000 7f4302e7d640 10 monclient: init 2024-03-25T19:26:44.491 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.490+0000 7f4302e7d640 10 monclient: _reopen_session rank -1 2024-03-25T19:26:44.491 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.490+0000 7f4302e7d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:26:44.491 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.490+0000 7f4302e7d640 10 monclient(hunting): picked mon.noname-a con 0x7f42fc0a98f0 addr v1:172.21.15.83:6789/0 2024-03-25T19:26:44.491 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.491+0000 7f4302e7d640 10 monclient(hunting): picked mon.noname-b con 0x7f42fc13f710 addr v1:172.21.15.83:6790/0 2024-03-25T19:26:44.491 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.491+0000 7f4302e7d640 10 monclient(hunting): picked mon.noname-c con 0x7f42fc1430c0 addr v1:172.21.15.103:6789/0 2024-03-25T19:26:44.492 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.491+0000 7f4302e7d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42fc142520 con 0x7f42fc0a98f0 2024-03-25T19:26:44.492 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.491+0000 7f4302e7d640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42fc0b28a0 con 0x7f42fc1430c0 2024-03-25T19:26:44.492 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.491+0000 7f4302e7d640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42fc0b2a30 con 0x7f42fc13f710 2024-03-25T19:26:44.492 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.491+0000 7f4302e7d640 10 monclient(hunting): _renew_subs 2024-03-25T19:26:44.492 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.491+0000 7f4302e7d640 10 monclient(hunting): authenticate will time out at 6461.762907s 2024-03-25T19:26:44.499 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.499+0000 7f42fbfff640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f42fc13f710 0x7f42fc1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:42256/0 (socket says 172.21.15.83:42256) 2024-03-25T19:26:44.499 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.499+0000 7f42fbfff640 1 -- 172.21.15.83:0/52960603 learned_addr learned my addr 172.21.15.83:0/52960603 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:26:44.517 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.517+0000 7f42fb7fe640 1 -- 172.21.15.83:0/52960603 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3049738572 0 0) 0x7f42fc142520 con 0x7f42fc0a98f0 2024-03-25T19:26:44.517 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.517+0000 7f42fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:26:44.517 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.517+0000 7f42fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:44.517 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.517+0000 7f42fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:44.517 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.517+0000 7f42fb7fe640 10 monclient(hunting): my global_id is 4536 2024-03-25T19:26:44.517 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.517+0000 7f42fb7fe640 1 -- 172.21.15.83:0/52960603 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42ec003520 con 0x7f42fc0a98f0 2024-03-25T19:26:44.522 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.522+0000 7f42fb7fe640 1 -- 172.21.15.83:0/52960603 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3406250807 0 0) 0x7f42fc0b28a0 con 0x7f42fc1430c0 2024-03-25T19:26:44.522 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.522+0000 7f42fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:26:44.522 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.522+0000 7f42fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:44.522 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.522+0000 7f42fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:44.522 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.522+0000 7f42fb7fe640 10 monclient(hunting): my global_id is 4546 2024-03-25T19:26:44.523 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.522+0000 7f42fb7fe640 1 -- 172.21.15.83:0/52960603 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42fc142520 con 0x7f42fc1430c0 2024-03-25T19:26:44.525 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.525+0000 7f42fb7fe640 1 -- 172.21.15.83:0/52960603 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1286578813 0 0) 0x7f42fc0b2a30 con 0x7f42fc13f710 2024-03-25T19:26:44.525 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.525+0000 7f42fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:26:44.525 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.525+0000 7f42fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:44.525 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.525+0000 7f42fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:44.525 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.525+0000 7f42fb7fe640 10 monclient(hunting): my global_id is 4553 2024-03-25T19:26:44.525 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.525+0000 7f42fb7fe640 1 -- 172.21.15.83:0/52960603 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42fc0b28a0 con 0x7f42fc13f710 2024-03-25T19:26:44.535 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.535+0000 7f42fb7fe640 1 -- 172.21.15.83:0/52960603 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2002871588 0 0) 0x7f42fc0b28a0 con 0x7f42fc13f710 2024-03-25T19:26:44.535 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.535+0000 7f42fb7fe640 1 -- 172.21.15.83:0/52960603 >> v1:172.21.15.103:6789/0 conn(0x7f42fc1430c0 legacy=0x7f42fc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:44.535 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.535+0000 7f42fb7fe640 1 -- 172.21.15.83:0/52960603 >> v1:172.21.15.83:6789/0 conn(0x7f42fc0a98f0 legacy=0x7f42fc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:44.535 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.535+0000 7f42fb7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:26:44.535 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.535+0000 7f42fb7fe640 1 monclient: found mon.noname-b 2024-03-25T19:26:44.535 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.535+0000 7f42fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:26:44.535 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.535+0000 7f42fb7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.83:6790/0 2024-03-25T19:26:44.535 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.535+0000 7f42fb7fe640 1 -- 172.21.15.83:0/52960603 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42fc0b44c0 con 0x7f42fc13f710 2024-03-25T19:26:44.535 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.535+0000 7f42fb7fe640 10 monclient: _finish_auth 0 2024-03-25T19:26:44.535 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.535+0000 7f42fb7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:26:44.535 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.535+0000 7f42fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:26:44.535 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.535+0000 7f4302e7d640 5 monclient: authenticate success, global_id 4553 2024-03-25T19:26:44.536 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.535+0000 7f4302e7d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:26:44.540 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.540+0000 7f42fb7fe640 1 -- 172.21.15.83:0/52960603 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f42e80014f0 con 0x7f42fc13f710 2024-03-25T19:26:44.540 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.540+0000 7f42fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:44.540 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.540+0000 7f42fb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:26:44.540 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.540+0000 7f42fb7fe640 10 monclient: dump: 2024-03-25T19:26:44.540 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:26:44.540 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:26:44.540 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:44.540 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:44.540 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:44.540 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:26:44.540 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:26:44.540 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:26:44.541 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:26:44.541 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:44.541 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.540+0000 7f4302e7d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:26:44.542 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.543+0000 7f42fb7fe640 1 -- 172.21.15.83:0/52960603 <== mon.2 v1:172.21.15.83:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f42e80017c0 con 0x7f42fc13f710 2024-03-25T19:26:44.543 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.543+0000 7f42fb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:26:44.543 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.543+0000 7f42fb7fe640 1 -- 172.21.15.83:0/52960603 <== mon.2 v1:172.21.15.83:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f42e8001b50 con 0x7f42fc13f710 2024-03-25T19:26:44.543 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.543+0000 7f42fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:44.543 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.543+0000 7f42fb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:26:44.543 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.543+0000 7f42fb7fe640 10 monclient: dump: 2024-03-25T19:26:44.543 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:26:44.543 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:26:44.543 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:44.543 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:44.543 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:44.543 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:26:44.543 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:26:44.543 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:26:44.543 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:26:44.543 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:44.544 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.543+0000 7f4302e7d640 10 monclient: get_monmap_and_config success 2024-03-25T19:26:44.544 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.543+0000 7f4302e7d640 4 set_mon_vals no callback set 2024-03-25T19:26:44.545 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.543+0000 7f4302e7d640 10 monclient: shutdown 2024-03-25T19:26:44.545 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.543+0000 7f4302e7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:26:44.545 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.543+0000 7f4302e7d640 1 -- 172.21.15.83:0/52960603 >> v1:172.21.15.83:6790/0 conn(0x7f42fc13f710 legacy=0x7f42fc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:44.545 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.543+0000 7f4302e7d640 1 -- 172.21.15.83:0/52960603 shutdown_connections 2024-03-25T19:26:44.545 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.544+0000 7f4302e7d640 1 -- 172.21.15.83:0/52960603 >> 172.21.15.83:0/52960603 conn(0x7f42fc0b7ea0 msgr2=0x7f42fc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:26:44.545 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.544+0000 7f4302e7d640 1 -- 172.21.15.83:0/52960603 shutdown_connections 2024-03-25T19:26:44.545 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.544+0000 7f4302e7d640 1 -- 172.21.15.83:0/52960603 wait complete. 2024-03-25T19:26:44.545 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.545+0000 7f4302e7d640 10 monclient: build_initial_monmap 2024-03-25T19:26:44.546 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.545+0000 7f4302e7d640 10 monclient: monmap: 2024-03-25T19:26:44.546 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:26:44.546 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:26:44.546 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:26:44.546 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:26:44.546 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:26:44.546 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:26:44.546 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:26:44.546 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:26:44.546 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:26:44.546 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:44.546 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.545+0000 7f4302e7d640 1 librados: starting msgr at 2024-03-25T19:26:44.546 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.545+0000 7f4302e7d640 1 librados: starting objecter 2024-03-25T19:26:44.546 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.545+0000 7f4302e7d640 1 Processor -- start 2024-03-25T19:26:44.547 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.545+0000 7f4302e7d640 1 -- start start 2024-03-25T19:26:44.547 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.545+0000 7f4302e7d640 1 librados: setting wanted keys 2024-03-25T19:26:44.547 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.545+0000 7f4302e7d640 1 librados: calling monclient init 2024-03-25T19:26:44.547 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.545+0000 7f4302e7d640 10 monclient: init 2024-03-25T19:26:44.547 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.546+0000 7f4302e7d640 10 monclient: _reopen_session rank -1 2024-03-25T19:26:44.548 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.546+0000 7f4302e7d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:26:44.548 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.546+0000 7f4302e7d640 10 monclient(hunting): picked mon.noname-c con 0x7f42fc0a98f0 addr v1:172.21.15.83:6790/0 2024-03-25T19:26:44.548 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.546+0000 7f4302e7d640 10 monclient(hunting): picked mon.noname-a con 0x7f42fc13f710 addr v1:172.21.15.83:6789/0 2024-03-25T19:26:44.548 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.546+0000 7f4302e7d640 10 monclient(hunting): picked mon.noname-b con 0x7f42fc1430c0 addr v1:172.21.15.103:6789/0 2024-03-25T19:26:44.548 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.546+0000 7f4302e7d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42fc14ee50 con 0x7f42fc13f710 2024-03-25T19:26:44.548 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.546+0000 7f4302e7d640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42fc14e410 con 0x7f42fc1430c0 2024-03-25T19:26:44.548 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.546+0000 7f4302e7d640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42fc14d830 con 0x7f42fc0a98f0 2024-03-25T19:26:44.548 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.546+0000 7f4302e7d640 10 monclient(hunting): _renew_subs 2024-03-25T19:26:44.548 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.546+0000 7f4302e7d640 10 monclient(hunting): authenticate will time out at 6461.818336s 2024-03-25T19:26:44.553 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.553+0000 7f430141a640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f42fc1430c0 0x7f42fc1e7350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:49510/0 (socket says 172.21.15.83:49510) 2024-03-25T19:26:44.553 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.553+0000 7f430141a640 1 -- 172.21.15.83:0/4256844319 learned_addr learned my addr 172.21.15.83:0/4256844319 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:26:44.557 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.557+0000 7f42f97fa640 10 client.?.objecter ms_handle_connect 0x7f42fc1430c0 2024-03-25T19:26:44.557 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.557+0000 7f42f97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:44.566 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.567+0000 7f42f97fa640 10 client.?.objecter ms_handle_connect 0x7f42fc0a98f0 2024-03-25T19:26:44.567 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.567+0000 7f42f97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:44.572 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.572+0000 7f42f97fa640 1 -- 172.21.15.83:0/4256844319 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 751617092 0 0) 0x7f42fc14e410 con 0x7f42fc1430c0 2024-03-25T19:26:44.572 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.572+0000 7f42f97fa640 10 monclient(hunting): _negotiate 2024-03-25T19:26:44.572 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.572+0000 7f42f97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:44.572 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.572+0000 7f42f97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:44.572 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.572+0000 7f42f97fa640 10 monclient(hunting): my global_id is 4549 2024-03-25T19:26:44.572 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.572+0000 7f42f97fa640 1 -- 172.21.15.83:0/4256844319 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42e0004eb0 con 0x7f42fc1430c0 2024-03-25T19:26:44.575 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.575+0000 7f42f97fa640 10 client.?.objecter ms_handle_connect 0x7f42fc13f710 2024-03-25T19:26:44.575 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.575+0000 7f42f97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:44.583 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.584+0000 7f42f97fa640 1 -- 172.21.15.83:0/4256844319 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1353925509 0 0) 0x7f42e0004eb0 con 0x7f42fc1430c0 2024-03-25T19:26:44.584 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.584+0000 7f42f97fa640 1 -- 172.21.15.83:0/4256844319 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42fc14e410 con 0x7f42fc1430c0 2024-03-25T19:26:44.584 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.584+0000 7f42f97fa640 1 -- 172.21.15.83:0/4256844319 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f42f4001bf0 con 0x7f42fc1430c0 2024-03-25T19:26:44.584 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.584+0000 7f42f97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:44.584 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.584+0000 7f42f97fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:26:44.584 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.584+0000 7f42f97fa640 10 monclient(hunting): dump: 2024-03-25T19:26:44.584 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:26:44.584 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:26:44.584 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:44.584 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:44.584 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:44.584 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:26:44.584 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:26:44.584 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:26:44.584 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:26:44.584 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:44.585 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.584+0000 7f42f97fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:26:44.586 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.586+0000 7f42f97fa640 1 -- 172.21.15.83:0/4256844319 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2489200025 0 0) 0x7f42fc14d830 con 0x7f42fc0a98f0 2024-03-25T19:26:44.586 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.586+0000 7f42f97fa640 10 monclient(hunting): _negotiate 2024-03-25T19:26:44.586 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.586+0000 7f42f97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:44.587 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.586+0000 7f42f97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:44.587 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.586+0000 7f42f97fa640 10 monclient(hunting): my global_id is 4556 2024-03-25T19:26:44.587 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.586+0000 7f42f97fa640 1 -- 172.21.15.83:0/4256844319 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42e0004eb0 con 0x7f42fc0a98f0 2024-03-25T19:26:44.587 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.586+0000 7f42f97fa640 1 -- 172.21.15.83:0/4256844319 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2376020462 0 0) 0x7f42fc14ee50 con 0x7f42fc13f710 2024-03-25T19:26:44.587 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.586+0000 7f42f97fa640 10 monclient(hunting): _negotiate 2024-03-25T19:26:44.587 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.586+0000 7f42f97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:44.587 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.586+0000 7f42f97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:44.587 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.586+0000 7f42f97fa640 10 monclient(hunting): my global_id is 4539 2024-03-25T19:26:44.587 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.586+0000 7f42f97fa640 1 -- 172.21.15.83:0/4256844319 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42fc14d830 con 0x7f42fc13f710 2024-03-25T19:26:44.597 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.597+0000 7f42f97fa640 1 -- 172.21.15.83:0/4256844319 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2187155051 0 0) 0x7f42fc14d830 con 0x7f42fc13f710 2024-03-25T19:26:44.597 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.598+0000 7f42f97fa640 1 -- 172.21.15.83:0/4256844319 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42fc14ee50 con 0x7f42fc13f710 2024-03-25T19:26:44.598 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.598+0000 7f42f97fa640 1 -- 172.21.15.83:0/4256844319 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f42e80014f0 con 0x7f42fc13f710 2024-03-25T19:26:44.598 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.598+0000 7f42f97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:44.598 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.598+0000 7f42f97fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:26:44.598 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.598+0000 7f42f97fa640 10 monclient(hunting): dump: 2024-03-25T19:26:44.598 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:26:44.598 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:26:44.598 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:44.598 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:44.598 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:44.598 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:26:44.598 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:26:44.598 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:26:44.598 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:26:44.598 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:44.605 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.605+0000 7f42f97fa640 1 -- 172.21.15.83:0/4256844319 <== mon.1 v1:172.21.15.103:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1562729965 0 0) 0x7f42fc14e410 con 0x7f42fc1430c0 2024-03-25T19:26:44.605 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.605+0000 7f42f97fa640 1 -- 172.21.15.83:0/4256844319 >> v1:172.21.15.83:6790/0 conn(0x7f42fc0a98f0 legacy=0x7f42fc1370d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:44.606 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.605+0000 7f42f97fa640 1 -- 172.21.15.83:0/4256844319 >> v1:172.21.15.83:6789/0 conn(0x7f42fc13f710 legacy=0x7f42fc1defb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:44.606 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.605+0000 7f42f97fa640 10 monclient: _finish_hunting 0 2024-03-25T19:26:44.606 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.605+0000 7f42f97fa640 1 monclient: found mon.b 2024-03-25T19:26:44.606 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.605+0000 7f42f97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:26:44.606 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.605+0000 7f42f97fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:26:44.606 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.605+0000 7f42f97fa640 1 -- 172.21.15.83:0/4256844319 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42fc14d9c0 con 0x7f42fc1430c0 2024-03-25T19:26:44.606 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.605+0000 7f42f97fa640 10 monclient: _finish_auth 0 2024-03-25T19:26:44.606 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.605+0000 7f42f97fa640 10 monclient: _check_auth_tickets 2024-03-25T19:26:44.606 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.605+0000 7f42f97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:26:44.606 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.606+0000 7f4302e7d640 5 monclient: authenticate success, global_id 4549 2024-03-25T19:26:44.606 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.606+0000 7f4302e7d640 10 monclient: _renew_subs 2024-03-25T19:26:44.606 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.606+0000 7f4302e7d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:26:44.606 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.606+0000 7f4302e7d640 1 -- 172.21.15.83:0/4256844319 --> v1:172.21.15.103:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42fc1377e0 con 0x7f42fc1430c0 2024-03-25T19:26:44.607 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.606+0000 7f4302e7d640 10 client.4549.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:26:44.607 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.606+0000 7f4302e7d640 10 monclient: _renew_subs 2024-03-25T19:26:44.607 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.606+0000 7f4302e7d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:26:44.607 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.606+0000 7f4302e7d640 1 -- 172.21.15.83:0/4256844319 --> v1:172.21.15.103:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f42fc14de30 con 0x7f42fc1430c0 2024-03-25T19:26:44.607 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.606+0000 7f4302e7d640 1 librados: init done 2024-03-25T19:26:44.607 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.607+0000 7f4302e7d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:44.607 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.607+0000 7f4302e7d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:44.607 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.607+0000 7f4302e7d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:26:44.607 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.607+0000 7f4302e7d640 1 -- 172.21.15.83:0/4256844319 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42fc14f320 con 0x7f42fc1430c0 2024-03-25T19:26:44.885 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.885+0000 7f42f97fa640 1 -- 172.21.15.83:0/4256844319 <== mon.1 v1:172.21.15.103:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f42f4003990 con 0x7f42fc1430c0 2024-03-25T19:26:44.885 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.885+0000 7f42f97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:26:44.885 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.885+0000 7f42f97fa640 1 -- 172.21.15.83:0/4256844319 <== mon.1 v1:172.21.15.103:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f42f4003da0 con 0x7f42fc1430c0 2024-03-25T19:26:44.885 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.885+0000 7f42f97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:44.885 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.885+0000 7f42f97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:26:44.886 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.885+0000 7f42f97fa640 10 monclient: dump: 2024-03-25T19:26:44.886 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:26:44.886 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:26:44.886 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:44.886 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:44.886 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:44.886 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:26:44.886 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:26:44.886 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:26:44.886 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:26:44.886 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:44.886 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.885+0000 7f42fa7fc640 4 set_mon_vals no callback set 2024-03-25T19:26:44.887 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.886+0000 7f42f97fa640 1 -- 172.21.15.83:0/4256844319 <== mon.1 v1:172.21.15.103:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7f42f4003fc0 con 0x7f42fc1430c0 2024-03-25T19:26:44.887 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.887+0000 7f4300c19640 10 monclient: get_auth_request con 0x7f42e006df80 auth_method 0 2024-03-25T19:26:44.887 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.887+0000 7f42f97fa640 10 client.4549.objecter ms_handle_connect 0x7f42e006df80 2024-03-25T19:26:44.892 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.892+0000 7f42f97fa640 1 -- 172.21.15.83:0/4256844319 <== mon.1 v1:172.21.15.103:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7f42f4054ab0 con 0x7f42fc1430c0 2024-03-25T19:26:44.892 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.892+0000 7f42f97fa640 10 client.4549.objecter ms_dispatch 0x7f42fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:26:44.892 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.892+0000 7f42f97fa640 3 client.4549.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:26:44.892 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.892+0000 7f42f97fa640 3 client.4549.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:26:44.892 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.892+0000 7f42f97fa640 20 client.4549.objecter dump_active .. 0 homeless 2024-03-25T19:26:44.892 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.893+0000 7f42f97fa640 1 -- 172.21.15.83:0/4256844319 <== mon.1 v1:172.21.15.103:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f42f40505c0 con 0x7f42fc1430c0 2024-03-25T19:26:44.892 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.893+0000 7f42f97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:44.892 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:44.893+0000 7f42f97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:26:45.130 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:45.130+0000 7f4302e7d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:45.130 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:45.130+0000 7f4302e7d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:45.130 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:45.130+0000 7f4302e7d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:26:45.131 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:45.130+0000 7f4302e7d640 1 -- 172.21.15.83:0/4256844319 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f42fc0627b0 con 0x7f42fc1430c0 2024-03-25T19:26:45.153 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:45.153+0000 7f42f97fa640 1 -- 172.21.15.83:0/4256844319 <== mon.1 v1:172.21.15.103:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7f42f40524f0 con 0x7f42fc1430c0 2024-03-25T19:26:45.153 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:45.153+0000 7f42f97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:45.153 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:45.153+0000 7f42f97fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:26:45.153 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:26:45.153 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:26:45.156 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:45.155+0000 7f4302e7d640 10 librados: watch_flush enter 2024-03-25T19:26:45.156 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:45.155+0000 7f4302e7d640 10 librados: watch_flush exit 2024-03-25T19:26:45.156 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:45.155+0000 7f4302e7d640 20 client.4549.objecter shutdown clearing up homeless session... 2024-03-25T19:26:45.156 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:45.155+0000 7f4302e7d640 10 client.4549.objecter successfully canceled tick 2024-03-25T19:26:45.156 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:45.156+0000 7f4302e7d640 1 -- 172.21.15.83:0/4256844319 >> v1:172.21.15.103:6800/345072327 conn(0x7f42e006df80 legacy=0x7f42e0070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:45.156 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:45.156+0000 7f4302e7d640 10 monclient: shutdown 2024-03-25T19:26:45.156 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:45.156+0000 7f4302e7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:26:45.156 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:45.156+0000 7f4302e7d640 1 -- 172.21.15.83:0/4256844319 >> v1:172.21.15.103:6789/0 conn(0x7f42fc1430c0 legacy=0x7f42fc1e7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:45.156 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:45.156+0000 7f4302e7d640 1 -- 172.21.15.83:0/4256844319 shutdown_connections 2024-03-25T19:26:45.156 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:45.156+0000 7f4302e7d640 1 -- 172.21.15.83:0/4256844319 >> 172.21.15.83:0/4256844319 conn(0x7f42fc0b7ea0 msgr2=0x7f42fc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:26:45.156 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:45.156+0000 7f4302e7d640 1 -- 172.21.15.83:0/4256844319 shutdown_connections 2024-03-25T19:26:45.157 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:45.156+0000 7f4302e7d640 1 -- 172.21.15.83:0/4256844319 wait complete. 2024-03-25T19:26:45.157 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:45.156+0000 7f4302e7d640 1 librados: shutdown 2024-03-25T19:26:45.169 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:26:48.418 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:07:30.345 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:48.424 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:07:30.351 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:48.472 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:07:30.399 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:48.477 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:07:30.387 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:48.486 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:07:30.396 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:48.494 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:07:30.422 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:48.502 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:07:30.430 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:48.506 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:07:30.416 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:48.532 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:07:30.459 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:51.975 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:26:52.107 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.107+0000 7f94a7c4e640 10 monclient: get_monmap_and_config 2024-03-25T19:26:52.107 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.107+0000 7f94a7c4e640 10 monclient: build_initial_monmap 2024-03-25T19:26:52.107 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.107+0000 7f94a7c4e640 10 monclient: monmap: 2024-03-25T19:26:52.107 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:26:52.107 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:26:52.107 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:26:52.108217+0000 2024-03-25T19:26:52.107 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:26:52.108217+0000 2024-03-25T19:26:52.107 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:26:52.107 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:26:52.107 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:26:52.107 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:26:52.107 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:26:52.108 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:52.109 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.107+0000 7f94a7c4e640 1 Processor -- start 2024-03-25T19:26:52.109 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.108+0000 7f94a7c4e640 1 -- start start 2024-03-25T19:26:52.109 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.108+0000 7f94a7c4e640 10 monclient: init 2024-03-25T19:26:52.109 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.108+0000 7f94a7c4e640 10 monclient: _reopen_session rank -1 2024-03-25T19:26:52.109 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.108+0000 7f94a7c4e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:26:52.109 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.108+0000 7f94a7c4e640 10 monclient(hunting): picked mon.noname-b con 0x7f94a00b4d60 addr v1:172.21.15.83:6790/0 2024-03-25T19:26:52.109 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.108+0000 7f94a7c4e640 10 monclient(hunting): picked mon.noname-c con 0x7f94a00b1730 addr v1:172.21.15.103:6789/0 2024-03-25T19:26:52.109 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.108+0000 7f94a7c4e640 10 monclient(hunting): picked mon.noname-a con 0x7f94a014be90 addr v1:172.21.15.83:6789/0 2024-03-25T19:26:52.109 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.108+0000 7f94a7c4e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94a00af940 con 0x7f94a014be90 2024-03-25T19:26:52.109 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.108+0000 7f94a7c4e640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94a00b0a70 con 0x7f94a00b1730 2024-03-25T19:26:52.109 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.108+0000 7f94a7c4e640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94a00b0d50 con 0x7f94a00b4d60 2024-03-25T19:26:52.109 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.108+0000 7f94a7c4e640 10 monclient(hunting): _renew_subs 2024-03-25T19:26:52.109 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.108+0000 7f94a7c4e640 10 monclient(hunting): authenticate will time out at 6469.380505s 2024-03-25T19:26:52.110 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.109+0000 7f94a51e9640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f94a00b1730 0x7f94a00b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:38382/0 (socket says 172.21.15.83:38382) 2024-03-25T19:26:52.110 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.109+0000 7f94a51e9640 1 -- 172.21.15.83:0/2025090673 learned_addr learned my addr 172.21.15.83:0/2025090673 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:26:52.144 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.144+0000 7f94a49e8640 1 -- 172.21.15.83:0/2025090673 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1957216700 0 0) 0x7f94a00b0d50 con 0x7f94a00b4d60 2024-03-25T19:26:52.144 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.144+0000 7f94a49e8640 10 monclient(hunting): _negotiate 2024-03-25T19:26:52.144 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.144+0000 7f94a49e8640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:52.144 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.144+0000 7f94a49e8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:52.144 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.144+0000 7f94a49e8640 10 monclient(hunting): my global_id is 4559 2024-03-25T19:26:52.144 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.144+0000 7f94a49e8640 1 -- 172.21.15.83:0/2025090673 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f948c003520 con 0x7f94a00b4d60 2024-03-25T19:26:52.145 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.145+0000 7f94a49e8640 1 -- 172.21.15.83:0/2025090673 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3247618460 0 0) 0x7f94a00af940 con 0x7f94a014be90 2024-03-25T19:26:52.145 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.145+0000 7f94a49e8640 10 monclient(hunting): _negotiate 2024-03-25T19:26:52.145 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.145+0000 7f94a49e8640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:52.145 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.145+0000 7f94a49e8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:52.145 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.145+0000 7f94a49e8640 10 monclient(hunting): my global_id is 4542 2024-03-25T19:26:52.145 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.145+0000 7f94a49e8640 1 -- 172.21.15.83:0/2025090673 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94a00b0d50 con 0x7f94a014be90 2024-03-25T19:26:52.160 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.160+0000 7f94a49e8640 1 -- 172.21.15.83:0/2025090673 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 79384891 0 0) 0x7f948c003520 con 0x7f94a00b4d60 2024-03-25T19:26:52.160 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.160+0000 7f94a49e8640 1 -- 172.21.15.83:0/2025090673 >> v1:172.21.15.103:6789/0 conn(0x7f94a00b1730 legacy=0x7f94a00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:52.161 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.161+0000 7f94a49e8640 1 -- 172.21.15.83:0/2025090673 >> v1:172.21.15.83:6789/0 conn(0x7f94a014be90 legacy=0x7f94a014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:52.161 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.161+0000 7f94a49e8640 10 monclient: _finish_hunting 0 2024-03-25T19:26:52.161 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.161+0000 7f94a49e8640 1 monclient: found mon.noname-b 2024-03-25T19:26:52.161 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.161+0000 7f94a49e8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:26:52.161 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.161+0000 7f94a49e8640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.83:6790/0 2024-03-25T19:26:52.161 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.161+0000 7f94a49e8640 1 -- 172.21.15.83:0/2025090673 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94a00b1030 con 0x7f94a00b4d60 2024-03-25T19:26:52.161 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.161+0000 7f94a49e8640 10 monclient: _finish_auth 0 2024-03-25T19:26:52.161 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.161+0000 7f94a49e8640 10 monclient: _check_auth_tickets 2024-03-25T19:26:52.161 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.161+0000 7f94a49e8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:26:52.161 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.161+0000 7f94a7c4e640 5 monclient: authenticate success, global_id 4559 2024-03-25T19:26:52.161 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.161+0000 7f94a49e8640 1 -- 172.21.15.83:0/2025090673 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f9490001690 con 0x7f94a00b4d60 2024-03-25T19:26:52.161 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.161+0000 7f94a49e8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:52.161 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.161+0000 7f94a49e8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:26:52.161 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.161+0000 7f94a49e8640 10 monclient: dump: 2024-03-25T19:26:52.161 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:26:52.161 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:26:52.162 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:52.162 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:52.162 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:52.162 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:26:52.162 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:26:52.162 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:26:52.162 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:26:52.162 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:52.162 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.161+0000 7f94a7c4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:26:52.187 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.187+0000 7f94a49e8640 1 -- 172.21.15.83:0/2025090673 <== mon.2 v1:172.21.15.83:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f94900017f0 con 0x7f94a00b4d60 2024-03-25T19:26:52.187 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.187+0000 7f94a49e8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:26:52.187 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.187+0000 7f94a49e8640 1 -- 172.21.15.83:0/2025090673 <== mon.2 v1:172.21.15.83:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f9490001c00 con 0x7f94a00b4d60 2024-03-25T19:26:52.188 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.187+0000 7f94a7c4e640 10 monclient: get_monmap_and_config success 2024-03-25T19:26:52.188 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.187+0000 7f94a7c4e640 4 set_mon_vals no callback set 2024-03-25T19:26:52.188 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.187+0000 7f94a49e8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:52.188 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.187+0000 7f94a49e8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:26:52.188 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.187+0000 7f94a49e8640 10 monclient: dump: 2024-03-25T19:26:52.188 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:26:52.188 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:26:52.188 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:52.188 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:52.188 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:52.188 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:26:52.188 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:26:52.188 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:26:52.188 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:26:52.188 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:52.191 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.188+0000 7f94a7c4e640 10 monclient: shutdown 2024-03-25T19:26:52.191 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.188+0000 7f94a7c4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:26:52.191 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.188+0000 7f94a7c4e640 1 -- 172.21.15.83:0/2025090673 >> v1:172.21.15.83:6790/0 conn(0x7f94a00b4d60 legacy=0x7f94a013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:52.191 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.188+0000 7f94a7c4e640 1 -- 172.21.15.83:0/2025090673 shutdown_connections 2024-03-25T19:26:52.191 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.188+0000 7f94a7c4e640 1 -- 172.21.15.83:0/2025090673 >> 172.21.15.83:0/2025090673 conn(0x7f94a00617a0 msgr2=0x7f94a0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:26:52.191 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.188+0000 7f94a7c4e640 1 -- 172.21.15.83:0/2025090673 shutdown_connections 2024-03-25T19:26:52.191 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.188+0000 7f94a7c4e640 1 -- 172.21.15.83:0/2025090673 wait complete. 2024-03-25T19:26:52.191 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.189+0000 7f94a7c4e640 10 monclient: build_initial_monmap 2024-03-25T19:26:52.191 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.189+0000 7f94a7c4e640 10 monclient: monmap: 2024-03-25T19:26:52.191 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:26:52.191 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:26:52.191 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:26:52.191 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:26:52.191 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:26:52.192 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:26:52.192 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:26:52.192 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:26:52.192 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:26:52.192 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:52.192 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.190+0000 7f94a7c4e640 1 librados: starting msgr at 2024-03-25T19:26:52.192 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.190+0000 7f94a7c4e640 1 librados: starting objecter 2024-03-25T19:26:52.192 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.190+0000 7f94a7c4e640 1 Processor -- start 2024-03-25T19:26:52.192 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.190+0000 7f94a7c4e640 1 -- start start 2024-03-25T19:26:52.192 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.190+0000 7f94a7c4e640 1 librados: setting wanted keys 2024-03-25T19:26:52.192 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.190+0000 7f94a7c4e640 1 librados: calling monclient init 2024-03-25T19:26:52.192 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.190+0000 7f94a7c4e640 10 monclient: init 2024-03-25T19:26:52.192 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.190+0000 7f94a7c4e640 10 monclient: _reopen_session rank -1 2024-03-25T19:26:52.192 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.190+0000 7f94a7c4e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:26:52.192 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.191+0000 7f94a7c4e640 10 monclient(hunting): picked mon.noname-b con 0x7f94a00b1730 addr v1:172.21.15.103:6789/0 2024-03-25T19:26:52.192 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.191+0000 7f94a7c4e640 10 monclient(hunting): picked mon.noname-c con 0x7f94a00b4d60 addr v1:172.21.15.83:6790/0 2024-03-25T19:26:52.193 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.191+0000 7f94a7c4e640 10 monclient(hunting): picked mon.noname-a con 0x7f94a014be90 addr v1:172.21.15.83:6789/0 2024-03-25T19:26:52.193 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.191+0000 7f94a7c4e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94a013d420 con 0x7f94a014be90 2024-03-25T19:26:52.193 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.191+0000 7f94a7c4e640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94a013d0f0 con 0x7f94a00b1730 2024-03-25T19:26:52.193 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.191+0000 7f94a7c4e640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94a01376e0 con 0x7f94a00b4d60 2024-03-25T19:26:52.193 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.191+0000 7f94a7c4e640 10 monclient(hunting): _renew_subs 2024-03-25T19:26:52.193 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.191+0000 7f94a7c4e640 10 monclient(hunting): authenticate will time out at 6469.463050s 2024-03-25T19:26:52.195 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.196+0000 7f94a61eb640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f94a014be90 0x7f94a013d8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:34792/0 (socket says 172.21.15.83:34792) 2024-03-25T19:26:52.195 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.196+0000 7f94a61eb640 1 -- 172.21.15.83:0/3227498134 learned_addr learned my addr 172.21.15.83:0/3227498134 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:26:52.196 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.196+0000 7f94967fc640 10 client.?.objecter ms_handle_connect 0x7f94a014be90 2024-03-25T19:26:52.196 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.196+0000 7f94967fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:52.209 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.209+0000 7f94967fc640 10 client.?.objecter ms_handle_connect 0x7f94a00b1730 2024-03-25T19:26:52.209 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.209+0000 7f94967fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:52.209 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.210+0000 7f94967fc640 10 client.?.objecter ms_handle_connect 0x7f94a00b4d60 2024-03-25T19:26:52.210 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.210+0000 7f94967fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:52.211 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.211+0000 7f94967fc640 1 -- 172.21.15.83:0/3227498134 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1240488851 0 0) 0x7f94a013d420 con 0x7f94a014be90 2024-03-25T19:26:52.211 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.211+0000 7f94967fc640 10 monclient(hunting): _negotiate 2024-03-25T19:26:52.211 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.211+0000 7f94967fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:52.211 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.211+0000 7f94967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:52.211 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.211+0000 7f94967fc640 10 monclient(hunting): my global_id is 4545 2024-03-25T19:26:52.211 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.211+0000 7f94967fc640 1 -- 172.21.15.83:0/3227498134 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9484004eb0 con 0x7f94a014be90 2024-03-25T19:26:52.222 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.223+0000 7f94967fc640 1 -- 172.21.15.83:0/3227498134 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1079960042 0 0) 0x7f94a013d0f0 con 0x7f94a00b1730 2024-03-25T19:26:52.222 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.223+0000 7f94967fc640 10 monclient(hunting): _negotiate 2024-03-25T19:26:52.222 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.223+0000 7f94967fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:52.223 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.223+0000 7f94967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:52.223 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.223+0000 7f94967fc640 10 monclient(hunting): my global_id is 4555 2024-03-25T19:26:52.223 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.223+0000 7f94967fc640 1 -- 172.21.15.83:0/3227498134 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94a013d420 con 0x7f94a00b1730 2024-03-25T19:26:52.226 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.226+0000 7f94967fc640 1 -- 172.21.15.83:0/3227498134 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2696287223 0 0) 0x7f94a01376e0 con 0x7f94a00b4d60 2024-03-25T19:26:52.226 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.226+0000 7f94967fc640 10 monclient(hunting): _negotiate 2024-03-25T19:26:52.226 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.226+0000 7f94967fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:52.226 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.226+0000 7f94967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:52.226 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.226+0000 7f94967fc640 10 monclient(hunting): my global_id is 4562 2024-03-25T19:26:52.226 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.226+0000 7f94967fc640 1 -- 172.21.15.83:0/3227498134 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94a013d0f0 con 0x7f94a00b4d60 2024-03-25T19:26:52.228 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.228+0000 7f94967fc640 1 -- 172.21.15.83:0/3227498134 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3749590146 0 0) 0x7f9484004eb0 con 0x7f94a014be90 2024-03-25T19:26:52.228 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.228+0000 7f94967fc640 1 -- 172.21.15.83:0/3227498134 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94a01376e0 con 0x7f94a014be90 2024-03-25T19:26:52.228 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.228+0000 7f94967fc640 1 -- 172.21.15.83:0/3227498134 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f9498001c90 con 0x7f94a014be90 2024-03-25T19:26:52.228 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.228+0000 7f94967fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:52.228 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.228+0000 7f94967fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:26:52.228 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.228+0000 7f94967fc640 10 monclient(hunting): dump: 2024-03-25T19:26:52.228 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:26:52.228 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:26:52.229 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:52.229 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:52.229 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:52.229 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:26:52.229 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:26:52.229 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:26:52.229 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:26:52.229 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:52.229 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.228+0000 7f94967fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:26:52.244 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.244+0000 7f94967fc640 1 -- 172.21.15.83:0/3227498134 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1466742353 0 0) 0x7f94a013d420 con 0x7f94a00b1730 2024-03-25T19:26:52.244 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.244+0000 7f94967fc640 1 -- 172.21.15.83:0/3227498134 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9484004eb0 con 0x7f94a00b1730 2024-03-25T19:26:52.244 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.244+0000 7f94967fc640 1 -- 172.21.15.83:0/3227498134 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f9490001b00 con 0x7f94a00b1730 2024-03-25T19:26:52.244 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.244+0000 7f94967fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:52.244 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.244+0000 7f94967fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:26:52.244 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.244+0000 7f94967fc640 10 monclient(hunting): dump: 2024-03-25T19:26:52.245 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:26:52.245 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:26:52.245 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:52.245 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:52.245 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:52.245 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:26:52.245 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:26:52.245 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:26:52.245 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:26:52.245 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:52.252 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.252+0000 7f94967fc640 1 -- 172.21.15.83:0/3227498134 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4158687907 0 0) 0x7f94a013d0f0 con 0x7f94a00b4d60 2024-03-25T19:26:52.252 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.252+0000 7f94967fc640 1 -- 172.21.15.83:0/3227498134 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94a013d420 con 0x7f94a00b4d60 2024-03-25T19:26:52.252 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.252+0000 7f94967fc640 1 -- 172.21.15.83:0/3227498134 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f949c001860 con 0x7f94a00b4d60 2024-03-25T19:26:52.252 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.252+0000 7f94967fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:52.252 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.252+0000 7f94967fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:26:52.252 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.252+0000 7f94967fc640 10 monclient(hunting): dump: 2024-03-25T19:26:52.252 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:26:52.252 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:26:52.252 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:52.252 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:52.252 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:52.252 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:26:52.252 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:26:52.252 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:26:52.253 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:26:52.253 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:52.255 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.254+0000 7f94967fc640 1 -- 172.21.15.83:0/3227498134 <== mon.1 v1:172.21.15.103:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 528125083 0 0) 0x7f9484004eb0 con 0x7f94a00b1730 2024-03-25T19:26:52.255 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.254+0000 7f94967fc640 1 -- 172.21.15.83:0/3227498134 >> v1:172.21.15.83:6790/0 conn(0x7f94a00b4d60 legacy=0x7f94a0142910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:52.255 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.254+0000 7f94967fc640 1 -- 172.21.15.83:0/3227498134 >> v1:172.21.15.83:6789/0 conn(0x7f94a014be90 legacy=0x7f94a013d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:52.255 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.254+0000 7f94967fc640 10 monclient: _finish_hunting 0 2024-03-25T19:26:52.255 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.254+0000 7f94967fc640 1 monclient: found mon.b 2024-03-25T19:26:52.255 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.254+0000 7f94967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:26:52.255 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.254+0000 7f94967fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:26:52.255 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.254+0000 7f94967fc640 1 -- 172.21.15.83:0/3227498134 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94a0137870 con 0x7f94a00b1730 2024-03-25T19:26:52.255 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.254+0000 7f94967fc640 10 monclient: _finish_auth 0 2024-03-25T19:26:52.255 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.254+0000 7f94967fc640 10 monclient: _check_auth_tickets 2024-03-25T19:26:52.255 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.254+0000 7f94967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:26:52.255 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.254+0000 7f94a7c4e640 5 monclient: authenticate success, global_id 4555 2024-03-25T19:26:52.255 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.254+0000 7f94a7c4e640 10 monclient: _renew_subs 2024-03-25T19:26:52.256 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.254+0000 7f94a7c4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:26:52.256 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.254+0000 7f94a7c4e640 1 -- 172.21.15.83:0/3227498134 --> v1:172.21.15.103:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94a0136880 con 0x7f94a00b1730 2024-03-25T19:26:52.256 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.254+0000 7f94a7c4e640 10 client.4555.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:26:52.256 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.254+0000 7f94a7c4e640 10 monclient: _renew_subs 2024-03-25T19:26:52.256 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.254+0000 7f94a7c4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:26:52.256 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.254+0000 7f94a7c4e640 1 -- 172.21.15.83:0/3227498134 --> v1:172.21.15.103:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f94a0136bd0 con 0x7f94a00b1730 2024-03-25T19:26:52.256 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.254+0000 7f94a7c4e640 1 librados: init done 2024-03-25T19:26:52.256 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.255+0000 7f94a7c4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:52.257 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.255+0000 7f94a7c4e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:52.257 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.255+0000 7f94a7c4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:26:52.257 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.255+0000 7f94a7c4e640 1 -- 172.21.15.83:0/3227498134 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94a00627b0 con 0x7f94a00b1730 2024-03-25T19:26:52.558 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.558+0000 7f94967fc640 1 -- 172.21.15.83:0/3227498134 <== mon.1 v1:172.21.15.103:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9490001210 con 0x7f94a00b1730 2024-03-25T19:26:52.559 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.558+0000 7f94967fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:26:52.559 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.558+0000 7f94967fc640 1 -- 172.21.15.83:0/3227498134 <== mon.1 v1:172.21.15.103:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f9490003a40 con 0x7f94a00b1730 2024-03-25T19:26:52.559 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.558+0000 7f94967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:52.559 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.558+0000 7f94967fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:26:52.559 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.558+0000 7f94967fc640 10 monclient: dump: 2024-03-25T19:26:52.559 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:26:52.559 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:26:52.559 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:52.559 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:52.559 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:52.559 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:26:52.559 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:26:52.560 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:26:52.560 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:26:52.560 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:52.560 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.558+0000 7f9497fff640 4 set_mon_vals no callback set 2024-03-25T19:26:52.560 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.559+0000 7f94967fc640 1 -- 172.21.15.83:0/3227498134 <== mon.1 v1:172.21.15.103:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7f9490019a10 con 0x7f94a00b1730 2024-03-25T19:26:52.561 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.560+0000 7f94a51e9640 10 monclient: get_auth_request con 0x7f948406e160 auth_method 0 2024-03-25T19:26:52.561 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.560+0000 7f94967fc640 10 client.4555.objecter ms_handle_connect 0x7f948406e160 2024-03-25T19:26:52.565 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.565+0000 7f94967fc640 1 -- 172.21.15.83:0/3227498134 <== mon.1 v1:172.21.15.103:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7f9490053660 con 0x7f94a00b1730 2024-03-25T19:26:52.565 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.565+0000 7f94967fc640 10 client.4555.objecter ms_dispatch 0x7f94a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:26:52.565 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.565+0000 7f94967fc640 3 client.4555.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:26:52.565 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.565+0000 7f94967fc640 3 client.4555.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:26:52.565 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.565+0000 7f94967fc640 20 client.4555.objecter dump_active .. 0 homeless 2024-03-25T19:26:52.565 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.565+0000 7f94967fc640 1 -- 172.21.15.83:0/3227498134 <== mon.1 v1:172.21.15.103:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f949004f170 con 0x7f94a00b1730 2024-03-25T19:26:52.565 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.565+0000 7f94967fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:52.565 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.565+0000 7f94967fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:26:52.816 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.816+0000 7f94a7c4e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:52.816 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.816+0000 7f94a7c4e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:52.816 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.816+0000 7f94a7c4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:26:52.816 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.816+0000 7f94a7c4e640 1 -- 172.21.15.83:0/3227498134 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94a00008d0 con 0x7f94a00b1730 2024-03-25T19:26:52.845 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.845+0000 7f94967fc640 1 -- 172.21.15.83:0/3227498134 <== mon.1 v1:172.21.15.103:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7f94900510a0 con 0x7f94a00b1730 2024-03-25T19:26:52.845 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.845+0000 7f94967fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:52.845 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.845+0000 7f94967fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:26:52.845 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:26:52.845 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:26:52.847 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.847+0000 7f94a7c4e640 10 librados: watch_flush enter 2024-03-25T19:26:52.847 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.847+0000 7f94a7c4e640 10 librados: watch_flush exit 2024-03-25T19:26:52.848 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.847+0000 7f94a7c4e640 20 client.4555.objecter shutdown clearing up homeless session... 2024-03-25T19:26:52.848 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.847+0000 7f94a7c4e640 10 client.4555.objecter successfully canceled tick 2024-03-25T19:26:52.848 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.847+0000 7f94a7c4e640 1 -- 172.21.15.83:0/3227498134 >> v1:172.21.15.103:6800/345072327 conn(0x7f948406e160 legacy=0x7f9484070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:52.848 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.847+0000 7f94a7c4e640 10 monclient: shutdown 2024-03-25T19:26:52.848 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.847+0000 7f94a7c4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:26:52.848 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.847+0000 7f94a7c4e640 1 -- 172.21.15.83:0/3227498134 >> v1:172.21.15.103:6789/0 conn(0x7f94a00b1730 legacy=0x7f94a0136fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:52.848 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.847+0000 7f94a7c4e640 1 -- 172.21.15.83:0/3227498134 shutdown_connections 2024-03-25T19:26:52.848 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.847+0000 7f94a7c4e640 1 -- 172.21.15.83:0/3227498134 >> 172.21.15.83:0/3227498134 conn(0x7f94a00617a0 msgr2=0x7f94a0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:26:52.848 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.847+0000 7f94a7c4e640 1 -- 172.21.15.83:0/3227498134 shutdown_connections 2024-03-25T19:26:52.848 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.847+0000 7f94a7c4e640 1 -- 172.21.15.83:0/3227498134 wait complete. 2024-03-25T19:26:52.848 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:52.847+0000 7f94a7c4e640 1 librados: shutdown 2024-03-25T19:26:52.860 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:26:59.666 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:26:59.781 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.781+0000 7f7c4ea9d640 10 monclient: get_monmap_and_config 2024-03-25T19:26:59.781 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.781+0000 7f7c4ea9d640 10 monclient: build_initial_monmap 2024-03-25T19:26:59.782 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.781+0000 7f7c4ea9d640 10 monclient: monmap: 2024-03-25T19:26:59.782 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:26:59.782 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:26:59.782 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:26:59.782985+0000 2024-03-25T19:26:59.782 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:26:59.782985+0000 2024-03-25T19:26:59.782 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:26:59.782 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:26:59.782 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:26:59.782 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:26:59.782 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:26:59.782 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:59.783 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.781+0000 7f7c4ea9d640 1 Processor -- start 2024-03-25T19:26:59.783 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.782+0000 7f7c4ea9d640 1 -- start start 2024-03-25T19:26:59.783 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.782+0000 7f7c4ea9d640 10 monclient: init 2024-03-25T19:26:59.783 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.782+0000 7f7c4ea9d640 10 monclient: _reopen_session rank -1 2024-03-25T19:26:59.783 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.782+0000 7f7c4ea9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:26:59.784 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.783+0000 7f7c4ea9d640 10 monclient(hunting): picked mon.noname-b con 0x7f7c480b4d60 addr v1:172.21.15.83:6790/0 2024-03-25T19:26:59.784 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.783+0000 7f7c4ea9d640 10 monclient(hunting): picked mon.noname-a con 0x7f7c480b1730 addr v1:172.21.15.83:6789/0 2024-03-25T19:26:59.784 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.783+0000 7f7c4ea9d640 10 monclient(hunting): picked mon.noname-c con 0x7f7c4814be90 addr v1:172.21.15.103:6789/0 2024-03-25T19:26:59.784 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.783+0000 7f7c4ea9d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c480af940 con 0x7f7c480b1730 2024-03-25T19:26:59.784 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.783+0000 7f7c4ea9d640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c480b0a70 con 0x7f7c4814be90 2024-03-25T19:26:59.784 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.783+0000 7f7c4ea9d640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c480b0d50 con 0x7f7c480b4d60 2024-03-25T19:26:59.784 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.783+0000 7f7c4ea9d640 10 monclient(hunting): _renew_subs 2024-03-25T19:26:59.784 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.783+0000 7f7c4ea9d640 10 monclient(hunting): authenticate will time out at 6477.055832s 2024-03-25T19:26:59.793 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.792+0000 7f7c4d03a640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f7c4814be90 0x7f7c4814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:46632/0 (socket says 172.21.15.83:46632) 2024-03-25T19:26:59.793 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.792+0000 7f7c4d03a640 1 -- 172.21.15.83:0/1350277446 learned_addr learned my addr 172.21.15.83:0/1350277446 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:26:59.808 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.807+0000 7f7c477fe640 1 -- 172.21.15.83:0/1350277446 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 893850466 0 0) 0x7f7c480af940 con 0x7f7c480b1730 2024-03-25T19:26:59.808 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.807+0000 7f7c477fe640 10 monclient(hunting): _negotiate 2024-03-25T19:26:59.808 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.807+0000 7f7c477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:59.808 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.807+0000 7f7c477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:59.808 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.807+0000 7f7c477fe640 10 monclient(hunting): my global_id is 4548 2024-03-25T19:26:59.808 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.807+0000 7f7c477fe640 1 -- 172.21.15.83:0/1350277446 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c38003520 con 0x7f7c480b1730 2024-03-25T19:26:59.809 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.809+0000 7f7c477fe640 1 -- 172.21.15.83:0/1350277446 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3951500594 0 0) 0x7f7c480b0a70 con 0x7f7c4814be90 2024-03-25T19:26:59.810 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.809+0000 7f7c477fe640 10 monclient(hunting): _negotiate 2024-03-25T19:26:59.810 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.809+0000 7f7c477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:59.810 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.809+0000 7f7c477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:59.810 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.809+0000 7f7c477fe640 10 monclient(hunting): my global_id is 4558 2024-03-25T19:26:59.810 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.809+0000 7f7c477fe640 1 -- 172.21.15.83:0/1350277446 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c480af940 con 0x7f7c4814be90 2024-03-25T19:26:59.833 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.832+0000 7f7c477fe640 1 -- 172.21.15.83:0/1350277446 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3372012351 0 0) 0x7f7c38003520 con 0x7f7c480b1730 2024-03-25T19:26:59.833 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.832+0000 7f7c477fe640 1 -- 172.21.15.83:0/1350277446 >> v1:172.21.15.83:6790/0 conn(0x7f7c480b4d60 legacy=0x7f7c4813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:59.833 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.832+0000 7f7c477fe640 1 -- 172.21.15.83:0/1350277446 >> v1:172.21.15.103:6789/0 conn(0x7f7c4814be90 legacy=0x7f7c4814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:59.833 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.832+0000 7f7c477fe640 10 monclient: _finish_hunting 0 2024-03-25T19:26:59.833 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.832+0000 7f7c477fe640 1 monclient: found mon.noname-a 2024-03-25T19:26:59.833 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.832+0000 7f7c477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:26:59.833 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.832+0000 7f7c477fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.83:6789/0 2024-03-25T19:26:59.833 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.832+0000 7f7c477fe640 1 -- 172.21.15.83:0/1350277446 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c480b1030 con 0x7f7c480b1730 2024-03-25T19:26:59.833 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.832+0000 7f7c477fe640 10 monclient: _finish_auth 0 2024-03-25T19:26:59.833 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.832+0000 7f7c477fe640 10 monclient: _check_auth_tickets 2024-03-25T19:26:59.834 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.832+0000 7f7c477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:26:59.834 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.832+0000 7f7c4ea9d640 5 monclient: authenticate success, global_id 4548 2024-03-25T19:26:59.834 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.832+0000 7f7c477fe640 1 -- 172.21.15.83:0/1350277446 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f7c34001690 con 0x7f7c480b1730 2024-03-25T19:26:59.834 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.832+0000 7f7c4ea9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:26:59.834 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.832+0000 7f7c477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:59.834 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.832+0000 7f7c477fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:26:59.834 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.832+0000 7f7c477fe640 10 monclient: dump: 2024-03-25T19:26:59.834 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:26:59.834 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:26:59.834 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:59.834 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:59.834 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:59.834 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:26:59.834 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:26:59.834 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:26:59.835 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:26:59.835 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:59.835 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.832+0000 7f7c4ea9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:26:59.851 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.850+0000 7f7c477fe640 1 -- 172.21.15.83:0/1350277446 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7c340017f0 con 0x7f7c480b1730 2024-03-25T19:26:59.852 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.850+0000 7f7c477fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:26:59.852 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.850+0000 7f7c477fe640 1 -- 172.21.15.83:0/1350277446 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f7c34001c00 con 0x7f7c480b1730 2024-03-25T19:26:59.852 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.850+0000 7f7c4ea9d640 10 monclient: get_monmap_and_config success 2024-03-25T19:26:59.852 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.850+0000 7f7c4ea9d640 4 set_mon_vals no callback set 2024-03-25T19:26:59.852 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.850+0000 7f7c477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:59.852 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.850+0000 7f7c477fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:26:59.852 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.850+0000 7f7c477fe640 10 monclient: dump: 2024-03-25T19:26:59.852 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:26:59.852 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:26:59.852 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:59.852 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:59.852 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:59.852 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:26:59.853 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:26:59.853 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:26:59.853 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:26:59.853 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:59.854 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.851+0000 7f7c4ea9d640 10 monclient: shutdown 2024-03-25T19:26:59.854 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.851+0000 7f7c4ea9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:26:59.854 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.851+0000 7f7c4ea9d640 1 -- 172.21.15.83:0/1350277446 >> v1:172.21.15.83:6789/0 conn(0x7f7c480b1730 legacy=0x7f7c480b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:59.854 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.851+0000 7f7c4ea9d640 1 -- 172.21.15.83:0/1350277446 shutdown_connections 2024-03-25T19:26:59.854 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.851+0000 7f7c4ea9d640 1 -- 172.21.15.83:0/1350277446 >> 172.21.15.83:0/1350277446 conn(0x7f7c480617a0 msgr2=0x7f7c48061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:26:59.854 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.851+0000 7f7c4ea9d640 1 -- 172.21.15.83:0/1350277446 shutdown_connections 2024-03-25T19:26:59.854 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.852+0000 7f7c4ea9d640 1 -- 172.21.15.83:0/1350277446 wait complete. 2024-03-25T19:26:59.854 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.852+0000 7f7c4ea9d640 10 monclient: build_initial_monmap 2024-03-25T19:26:59.854 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.852+0000 7f7c4ea9d640 10 monclient: monmap: 2024-03-25T19:26:59.855 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:26:59.855 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:26:59.855 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:26:59.855 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:26:59.855 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:26:59.855 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:26:59.855 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:26:59.855 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:26:59.855 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:26:59.855 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:59.855 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.853+0000 7f7c4ea9d640 1 librados: starting msgr at 2024-03-25T19:26:59.855 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.853+0000 7f7c4ea9d640 1 librados: starting objecter 2024-03-25T19:26:59.855 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.853+0000 7f7c4ea9d640 1 Processor -- start 2024-03-25T19:26:59.855 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.853+0000 7f7c4ea9d640 1 -- start start 2024-03-25T19:26:59.855 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.853+0000 7f7c4ea9d640 1 librados: setting wanted keys 2024-03-25T19:26:59.856 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.853+0000 7f7c4ea9d640 1 librados: calling monclient init 2024-03-25T19:26:59.856 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.853+0000 7f7c4ea9d640 10 monclient: init 2024-03-25T19:26:59.857 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.854+0000 7f7c4ea9d640 10 monclient: _reopen_session rank -1 2024-03-25T19:26:59.857 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.854+0000 7f7c4ea9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:26:59.857 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.854+0000 7f7c4ea9d640 10 monclient(hunting): picked mon.noname-a con 0x7f7c480b1730 addr v1:172.21.15.83:6789/0 2024-03-25T19:26:59.857 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.854+0000 7f7c4ea9d640 10 monclient(hunting): picked mon.noname-c con 0x7f7c480b4d60 addr v1:172.21.15.83:6790/0 2024-03-25T19:26:59.857 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.854+0000 7f7c4ea9d640 10 monclient(hunting): picked mon.noname-b con 0x7f7c4814be90 addr v1:172.21.15.103:6789/0 2024-03-25T19:26:59.857 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.854+0000 7f7c4ea9d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c481451e0 con 0x7f7c480b1730 2024-03-25T19:26:59.857 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.854+0000 7f7c4ea9d640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c481411f0 con 0x7f7c4814be90 2024-03-25T19:26:59.857 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.854+0000 7f7c4ea9d640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c48141380 con 0x7f7c480b4d60 2024-03-25T19:26:59.857 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.854+0000 7f7c4ea9d640 10 monclient(hunting): _renew_subs 2024-03-25T19:26:59.857 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.854+0000 7f7c4ea9d640 10 monclient(hunting): authenticate will time out at 6477.127312s 2024-03-25T19:26:59.863 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.862+0000 7f7c47fff640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f7c480b4d60 0x7f7c481427d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:54026/0 (socket says 172.21.15.83:54026) 2024-03-25T19:26:59.863 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.862+0000 7f7c47fff640 1 -- 172.21.15.83:0/3392691381 learned_addr learned my addr 172.21.15.83:0/3392691381 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:26:59.864 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.864+0000 7f7c457fa640 10 client.?.objecter ms_handle_connect 0x7f7c480b4d60 2024-03-25T19:26:59.865 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.864+0000 7f7c457fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:59.866 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.865+0000 7f7c457fa640 10 client.?.objecter ms_handle_connect 0x7f7c4814be90 2024-03-25T19:26:59.866 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.865+0000 7f7c457fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:59.869 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.868+0000 7f7c457fa640 10 client.?.objecter ms_handle_connect 0x7f7c480b1730 2024-03-25T19:26:59.869 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.868+0000 7f7c457fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:59.879 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.878+0000 7f7c457fa640 1 -- 172.21.15.83:0/3392691381 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4178253291 0 0) 0x7f7c481411f0 con 0x7f7c4814be90 2024-03-25T19:26:59.879 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.878+0000 7f7c457fa640 10 monclient(hunting): _negotiate 2024-03-25T19:26:59.879 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.878+0000 7f7c457fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:59.879 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.878+0000 7f7c457fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:59.879 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.878+0000 7f7c457fa640 10 monclient(hunting): my global_id is 4561 2024-03-25T19:26:59.880 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.878+0000 7f7c457fa640 1 -- 172.21.15.83:0/3392691381 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c2c004eb0 con 0x7f7c4814be90 2024-03-25T19:26:59.880 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.880+0000 7f7c457fa640 1 -- 172.21.15.83:0/3392691381 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 700045888 0 0) 0x7f7c48141380 con 0x7f7c480b4d60 2024-03-25T19:26:59.880 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.880+0000 7f7c457fa640 10 monclient(hunting): _negotiate 2024-03-25T19:26:59.880 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.880+0000 7f7c457fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:59.880 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.880+0000 7f7c457fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:59.880 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.880+0000 7f7c457fa640 10 monclient(hunting): my global_id is 4568 2024-03-25T19:26:59.880 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.880+0000 7f7c457fa640 1 -- 172.21.15.83:0/3392691381 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c481411f0 con 0x7f7c480b4d60 2024-03-25T19:26:59.883 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.883+0000 7f7c457fa640 1 -- 172.21.15.83:0/3392691381 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 390925811 0 0) 0x7f7c481451e0 con 0x7f7c480b1730 2024-03-25T19:26:59.883 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.883+0000 7f7c457fa640 10 monclient(hunting): _negotiate 2024-03-25T19:26:59.883 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.883+0000 7f7c457fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:59.883 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.883+0000 7f7c457fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:59.884 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.883+0000 7f7c457fa640 10 monclient(hunting): my global_id is 4551 2024-03-25T19:26:59.884 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.883+0000 7f7c457fa640 1 -- 172.21.15.83:0/3392691381 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c48141380 con 0x7f7c480b1730 2024-03-25T19:26:59.897 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.896+0000 7f7c457fa640 1 -- 172.21.15.83:0/3392691381 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3380463001 0 0) 0x7f7c481411f0 con 0x7f7c480b4d60 2024-03-25T19:26:59.897 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.896+0000 7f7c457fa640 1 -- 172.21.15.83:0/3392691381 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c481451e0 con 0x7f7c480b4d60 2024-03-25T19:26:59.897 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.896+0000 7f7c457fa640 1 -- 172.21.15.83:0/3392691381 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f7c34001b40 con 0x7f7c480b4d60 2024-03-25T19:26:59.897 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.896+0000 7f7c457fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:59.897 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.896+0000 7f7c457fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:26:59.897 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.896+0000 7f7c457fa640 10 monclient(hunting): dump: 2024-03-25T19:26:59.897 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:26:59.897 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:26:59.897 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:59.897 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:59.897 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:59.898 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:26:59.898 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:26:59.898 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:26:59.898 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:26:59.898 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:59.898 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.896+0000 7f7c457fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:26:59.898 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.897+0000 7f7c457fa640 1 -- 172.21.15.83:0/3392691381 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 155953280 0 0) 0x7f7c2c004eb0 con 0x7f7c4814be90 2024-03-25T19:26:59.898 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.897+0000 7f7c457fa640 1 -- 172.21.15.83:0/3392691381 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c481411f0 con 0x7f7c4814be90 2024-03-25T19:26:59.899 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.897+0000 7f7c457fa640 1 -- 172.21.15.83:0/3392691381 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f7c40001bd0 con 0x7f7c4814be90 2024-03-25T19:26:59.899 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.897+0000 7f7c457fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:59.899 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.897+0000 7f7c457fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:26:59.899 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.897+0000 7f7c457fa640 10 monclient(hunting): dump: 2024-03-25T19:26:59.899 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:26:59.899 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:26:59.899 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:59.899 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:26:59.899 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:59.899 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:26:59.899 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:26:59.899 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:26:59.899 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:26:59.899 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:26:59.903 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.902+0000 7f7c457fa640 1 -- 172.21.15.83:0/3392691381 <== mon.1 v1:172.21.15.103:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1635086554 0 0) 0x7f7c481411f0 con 0x7f7c4814be90 2024-03-25T19:26:59.903 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.902+0000 7f7c457fa640 1 -- 172.21.15.83:0/3392691381 >> v1:172.21.15.83:6790/0 conn(0x7f7c480b4d60 legacy=0x7f7c481427d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:59.903 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.902+0000 7f7c457fa640 1 -- 172.21.15.83:0/3392691381 >> v1:172.21.15.83:6789/0 conn(0x7f7c480b1730 legacy=0x7f7c48140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:59.903 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.902+0000 7f7c457fa640 10 monclient: _finish_hunting 0 2024-03-25T19:26:59.903 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.902+0000 7f7c457fa640 1 monclient: found mon.b 2024-03-25T19:26:59.904 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.902+0000 7f7c457fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:26:59.904 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.902+0000 7f7c457fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:26:59.904 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.902+0000 7f7c457fa640 1 -- 172.21.15.83:0/3392691381 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c48137eb0 con 0x7f7c4814be90 2024-03-25T19:26:59.904 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.902+0000 7f7c457fa640 10 monclient: _finish_auth 0 2024-03-25T19:26:59.904 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.902+0000 7f7c457fa640 10 monclient: _check_auth_tickets 2024-03-25T19:26:59.904 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.902+0000 7f7c457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:26:59.904 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.902+0000 7f7c4ea9d640 5 monclient: authenticate success, global_id 4561 2024-03-25T19:26:59.904 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.902+0000 7f7c4ea9d640 10 monclient: _renew_subs 2024-03-25T19:26:59.904 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.902+0000 7f7c4ea9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:26:59.904 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.902+0000 7f7c4ea9d640 1 -- 172.21.15.83:0/3392691381 --> v1:172.21.15.103:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c48138100 con 0x7f7c4814be90 2024-03-25T19:26:59.904 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.902+0000 7f7c4ea9d640 10 client.4561.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:26:59.904 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.902+0000 7f7c4ea9d640 10 monclient: _renew_subs 2024-03-25T19:26:59.904 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.902+0000 7f7c4ea9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:26:59.904 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.902+0000 7f7c4ea9d640 1 -- 172.21.15.83:0/3392691381 --> v1:172.21.15.103:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c481e3710 con 0x7f7c4814be90 2024-03-25T19:26:59.904 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.902+0000 7f7c4ea9d640 1 librados: init done 2024-03-25T19:26:59.905 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.903+0000 7f7c4ea9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:59.905 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.903+0000 7f7c4ea9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:59.905 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.903+0000 7f7c4ea9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:26:59.905 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:26:59.903+0000 7f7c4ea9d640 1 -- 172.21.15.83:0/3392691381 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c481453d0 con 0x7f7c4814be90 2024-03-25T19:27:00.179 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:00.178+0000 7f7c457fa640 1 -- 172.21.15.83:0/3392691381 <== mon.1 v1:172.21.15.103:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7c40002060 con 0x7f7c4814be90 2024-03-25T19:27:00.179 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:00.178+0000 7f7c457fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:00.180 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:00.178+0000 7f7c457fa640 1 -- 172.21.15.83:0/3392691381 <== mon.1 v1:172.21.15.103:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f7c40004d00 con 0x7f7c4814be90 2024-03-25T19:27:00.180 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:00.178+0000 7f7c457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:00.180 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:00.178+0000 7f7c457fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:27:00.180 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:00.178+0000 7f7c457fa640 10 monclient: dump: 2024-03-25T19:27:00.180 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:27:00.180 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:27:00.180 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:00.180 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:00.180 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:00.180 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:27:00.180 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:27:00.180 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:27:00.180 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:27:00.180 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:27:00.180 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:00.178+0000 7f7c46ffd640 4 set_mon_vals no callback set 2024-03-25T19:27:00.181 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:00.179+0000 7f7c457fa640 1 -- 172.21.15.83:0/3392691381 <== mon.1 v1:172.21.15.103:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7f7c4001acd0 con 0x7f7c4814be90 2024-03-25T19:27:00.181 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:00.180+0000 7f7c4c839640 10 monclient: get_auth_request con 0x7f7c2c06df70 auth_method 0 2024-03-25T19:27:00.181 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:00.180+0000 7f7c457fa640 10 client.4561.objecter ms_handle_connect 0x7f7c2c06df70 2024-03-25T19:27:00.188 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:00.187+0000 7f7c457fa640 1 -- 172.21.15.83:0/3392691381 <== mon.1 v1:172.21.15.103:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7f7c400549f0 con 0x7f7c4814be90 2024-03-25T19:27:00.188 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:00.187+0000 7f7c457fa640 10 client.4561.objecter ms_dispatch 0x7f7c48000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:27:00.188 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:00.187+0000 7f7c457fa640 3 client.4561.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:27:00.188 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:00.187+0000 7f7c457fa640 3 client.4561.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:27:00.188 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:00.187+0000 7f7c457fa640 20 client.4561.objecter dump_active .. 0 homeless 2024-03-25T19:27:00.189 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:00.189+0000 7f7c457fa640 1 -- 172.21.15.83:0/3392691381 <== mon.1 v1:172.21.15.103:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7c40050330 con 0x7f7c4814be90 2024-03-25T19:27:00.189 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:00.189+0000 7f7c457fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:00.189 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:00.189+0000 7f7c457fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:27:00.424 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:00.423+0000 7f7c4ea9d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:00.424 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:00.423+0000 7f7c4ea9d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:00.424 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:00.423+0000 7f7c4ea9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:27:00.424 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:00.423+0000 7f7c4ea9d640 1 -- 172.21.15.83:0/3392691381 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7c48145770 con 0x7f7c4814be90 2024-03-25T19:27:00.459 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:00.459+0000 7f7c457fa640 1 -- 172.21.15.83:0/3392691381 <== mon.1 v1:172.21.15.103:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7f7c48145770 con 0x7f7c4814be90 2024-03-25T19:27:00.459 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:00.459+0000 7f7c457fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:00.460 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:00.459+0000 7f7c457fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:27:00.460 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:27:00.460 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:27:00.462 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:00.461+0000 7f7c4ea9d640 10 librados: watch_flush enter 2024-03-25T19:27:00.462 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:00.461+0000 7f7c4ea9d640 10 librados: watch_flush exit 2024-03-25T19:27:00.462 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:00.461+0000 7f7c4ea9d640 20 client.4561.objecter shutdown clearing up homeless session... 2024-03-25T19:27:00.462 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:00.461+0000 7f7c4ea9d640 10 client.4561.objecter successfully canceled tick 2024-03-25T19:27:00.462 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:00.461+0000 7f7c4ea9d640 1 -- 172.21.15.83:0/3392691381 >> v1:172.21.15.103:6800/345072327 conn(0x7f7c2c06df70 legacy=0x7f7c2c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:00.462 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:00.461+0000 7f7c4ea9d640 10 monclient: shutdown 2024-03-25T19:27:00.462 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:00.461+0000 7f7c4ea9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:00.463 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:00.461+0000 7f7c4ea9d640 1 -- 172.21.15.83:0/3392691381 >> v1:172.21.15.103:6789/0 conn(0x7f7c4814be90 legacy=0x7f7c48142ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:00.463 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:00.461+0000 7f7c4ea9d640 1 -- 172.21.15.83:0/3392691381 shutdown_connections 2024-03-25T19:27:00.463 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:00.461+0000 7f7c4ea9d640 1 -- 172.21.15.83:0/3392691381 >> 172.21.15.83:0/3392691381 conn(0x7f7c480617a0 msgr2=0x7f7c4813c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:00.463 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:00.461+0000 7f7c4ea9d640 1 -- 172.21.15.83:0/3392691381 shutdown_connections 2024-03-25T19:27:00.463 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:00.461+0000 7f7c4ea9d640 1 -- 172.21.15.83:0/3392691381 wait complete. 2024-03-25T19:27:00.463 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:00.461+0000 7f7c4ea9d640 1 librados: shutdown 2024-03-25T19:27:00.473 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:27:07.279 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:27:07.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.388+0000 7f2a77460640 10 monclient: get_monmap_and_config 2024-03-25T19:27:07.388 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.388+0000 7f2a77460640 10 monclient: build_initial_monmap 2024-03-25T19:27:07.388 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.388+0000 7f2a77460640 10 monclient: monmap: 2024-03-25T19:27:07.388 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:27:07.388 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:27:07.388 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:27:07.389012+0000 2024-03-25T19:27:07.388 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:27:07.389012+0000 2024-03-25T19:27:07.388 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:07.388 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:27:07.388 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:27:07.388 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:27:07.388 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:27:07.388 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:27:07.390 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.388+0000 7f2a77460640 1 Processor -- start 2024-03-25T19:27:07.390 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.388+0000 7f2a77460640 1 -- start start 2024-03-25T19:27:07.390 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.388+0000 7f2a77460640 10 monclient: init 2024-03-25T19:27:07.390 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.389+0000 7f2a77460640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:07.390 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.389+0000 7f2a77460640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:27:07.390 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.389+0000 7f2a77460640 10 monclient(hunting): picked mon.noname-a con 0x7f2a7006ea40 addr v1:172.21.15.83:6789/0 2024-03-25T19:27:07.390 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.389+0000 7f2a77460640 10 monclient(hunting): picked mon.noname-b con 0x7f2a70071800 addr v1:172.21.15.83:6790/0 2024-03-25T19:27:07.390 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.389+0000 7f2a77460640 10 monclient(hunting): picked mon.noname-c con 0x7f2a70071c40 addr v1:172.21.15.103:6789/0 2024-03-25T19:27:07.390 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.389+0000 7f2a77460640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a7006f1f0 con 0x7f2a7006ea40 2024-03-25T19:27:07.390 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.389+0000 7f2a77460640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a70070d00 con 0x7f2a70071c40 2024-03-25T19:27:07.390 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.389+0000 7f2a77460640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a700720b0 con 0x7f2a70071800 2024-03-25T19:27:07.391 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.389+0000 7f2a77460640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:07.391 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.389+0000 7f2a77460640 10 monclient(hunting): authenticate will time out at 6484.661525s 2024-03-25T19:27:07.393 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.393+0000 7f2a759fd640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f2a70071c40 0x7f2a7014a7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:56762/0 (socket says 172.21.15.83:56762) 2024-03-25T19:27:07.393 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.393+0000 7f2a759fd640 1 -- 172.21.15.83:0/2826798327 learned_addr learned my addr 172.21.15.83:0/2826798327 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:27:07.405 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.405+0000 7f2a67fff640 1 -- 172.21.15.83:0/2826798327 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1139943113 0 0) 0x7f2a70070d00 con 0x7f2a70071c40 2024-03-25T19:27:07.405 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.405+0000 7f2a67fff640 10 monclient(hunting): _negotiate 2024-03-25T19:27:07.405 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.405+0000 7f2a67fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:07.405 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.405+0000 7f2a67fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:07.405 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.405+0000 7f2a67fff640 10 monclient(hunting): my global_id is 4564 2024-03-25T19:27:07.405 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.405+0000 7f2a67fff640 1 -- 172.21.15.83:0/2826798327 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a5c003520 con 0x7f2a70071c40 2024-03-25T19:27:07.409 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.409+0000 7f2a67fff640 1 -- 172.21.15.83:0/2826798327 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4016963739 0 0) 0x7f2a700720b0 con 0x7f2a70071800 2024-03-25T19:27:07.409 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.409+0000 7f2a67fff640 10 monclient(hunting): _negotiate 2024-03-25T19:27:07.409 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.409+0000 7f2a67fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:07.409 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.409+0000 7f2a67fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:07.409 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.409+0000 7f2a67fff640 10 monclient(hunting): my global_id is 4571 2024-03-25T19:27:07.409 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.409+0000 7f2a67fff640 1 -- 172.21.15.83:0/2826798327 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a70070d00 con 0x7f2a70071800 2024-03-25T19:27:07.414 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.415+0000 7f2a67fff640 1 -- 172.21.15.83:0/2826798327 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2554158036 0 0) 0x7f2a7006f1f0 con 0x7f2a7006ea40 2024-03-25T19:27:07.415 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.415+0000 7f2a67fff640 10 monclient(hunting): _negotiate 2024-03-25T19:27:07.415 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.415+0000 7f2a67fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:07.415 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.415+0000 7f2a67fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:07.415 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.415+0000 7f2a67fff640 10 monclient(hunting): my global_id is 4554 2024-03-25T19:27:07.415 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.415+0000 7f2a67fff640 1 -- 172.21.15.83:0/2826798327 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a700720b0 con 0x7f2a7006ea40 2024-03-25T19:27:07.417 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.416+0000 7f2a67fff640 1 -- 172.21.15.83:0/2826798327 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3604942819 0 0) 0x7f2a70070d00 con 0x7f2a70071800 2024-03-25T19:27:07.417 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.416+0000 7f2a67fff640 1 -- 172.21.15.83:0/2826798327 >> v1:172.21.15.103:6789/0 conn(0x7f2a70071c40 legacy=0x7f2a7014a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:07.417 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.416+0000 7f2a67fff640 1 -- 172.21.15.83:0/2826798327 >> v1:172.21.15.83:6789/0 conn(0x7f2a7006ea40 legacy=0x7f2a7013ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:07.417 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.416+0000 7f2a67fff640 10 monclient: _finish_hunting 0 2024-03-25T19:27:07.417 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.416+0000 7f2a67fff640 1 monclient: found mon.noname-b 2024-03-25T19:27:07.417 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.416+0000 7f2a67fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:07.417 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.416+0000 7f2a67fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.83:6790/0 2024-03-25T19:27:07.417 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.416+0000 7f2a67fff640 1 -- 172.21.15.83:0/2826798327 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a70072780 con 0x7f2a70071800 2024-03-25T19:27:07.417 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.416+0000 7f2a67fff640 10 monclient: _finish_auth 0 2024-03-25T19:27:07.417 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.416+0000 7f2a67fff640 10 monclient: _check_auth_tickets 2024-03-25T19:27:07.417 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.416+0000 7f2a67fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:07.417 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.416+0000 7f2a67fff640 1 -- 172.21.15.83:0/2826798327 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f2a60001670 con 0x7f2a70071800 2024-03-25T19:27:07.417 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.416+0000 7f2a77460640 5 monclient: authenticate success, global_id 4571 2024-03-25T19:27:07.417 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.416+0000 7f2a77460640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:27:07.417 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.416+0000 7f2a67fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:07.418 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.417+0000 7f2a67fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:27:07.418 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.417+0000 7f2a67fff640 10 monclient: dump: 2024-03-25T19:27:07.418 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:27:07.418 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:27:07.418 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:07.418 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:07.418 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:07.418 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:27:07.418 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:27:07.418 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:27:07.418 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:27:07.418 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:27:07.418 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.417+0000 7f2a77460640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:27:07.442 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.442+0000 7f2a67fff640 1 -- 172.21.15.83:0/2826798327 <== mon.2 v1:172.21.15.83:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a600017d0 con 0x7f2a70071800 2024-03-25T19:27:07.442 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.442+0000 7f2a67fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:07.442 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.442+0000 7f2a67fff640 1 -- 172.21.15.83:0/2826798327 <== mon.2 v1:172.21.15.83:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f2a60001be0 con 0x7f2a70071800 2024-03-25T19:27:07.442 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.442+0000 7f2a67fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:07.442 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.442+0000 7f2a67fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:27:07.442 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.442+0000 7f2a67fff640 10 monclient: dump: 2024-03-25T19:27:07.442 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:27:07.442 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:27:07.442 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:07.442 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:07.443 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:07.443 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:27:07.443 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:27:07.443 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:27:07.443 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:27:07.443 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:27:07.443 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.442+0000 7f2a77460640 10 monclient: get_monmap_and_config success 2024-03-25T19:27:07.443 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.442+0000 7f2a77460640 4 set_mon_vals no callback set 2024-03-25T19:27:07.444 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.443+0000 7f2a77460640 10 monclient: shutdown 2024-03-25T19:27:07.444 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.443+0000 7f2a77460640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:07.444 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.443+0000 7f2a77460640 1 -- 172.21.15.83:0/2826798327 >> v1:172.21.15.83:6790/0 conn(0x7f2a70071800 legacy=0x7f2a70147070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:07.445 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.443+0000 7f2a77460640 1 -- 172.21.15.83:0/2826798327 shutdown_connections 2024-03-25T19:27:07.445 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.443+0000 7f2a77460640 1 -- 172.21.15.83:0/2826798327 >> 172.21.15.83:0/2826798327 conn(0x7f2a7006b630 msgr2=0x7f2a700b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:07.445 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.443+0000 7f2a77460640 1 -- 172.21.15.83:0/2826798327 shutdown_connections 2024-03-25T19:27:07.445 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.443+0000 7f2a77460640 1 -- 172.21.15.83:0/2826798327 wait complete. 2024-03-25T19:27:07.445 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.444+0000 7f2a77460640 10 monclient: build_initial_monmap 2024-03-25T19:27:07.445 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.444+0000 7f2a77460640 10 monclient: monmap: 2024-03-25T19:27:07.445 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:27:07.445 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:27:07.445 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:27:07.445 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:27:07.445 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:07.445 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:27:07.445 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:27:07.445 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:27:07.445 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:27:07.446 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:27:07.446 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.444+0000 7f2a77460640 1 librados: starting msgr at 2024-03-25T19:27:07.446 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.444+0000 7f2a77460640 1 librados: starting objecter 2024-03-25T19:27:07.446 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.445+0000 7f2a77460640 1 Processor -- start 2024-03-25T19:27:07.446 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.445+0000 7f2a77460640 1 -- start start 2024-03-25T19:27:07.446 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.445+0000 7f2a77460640 1 librados: setting wanted keys 2024-03-25T19:27:07.446 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.445+0000 7f2a77460640 1 librados: calling monclient init 2024-03-25T19:27:07.446 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.445+0000 7f2a77460640 10 monclient: init 2024-03-25T19:27:07.447 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.445+0000 7f2a77460640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:07.447 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.445+0000 7f2a77460640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:27:07.447 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.445+0000 7f2a77460640 10 monclient(hunting): picked mon.noname-c con 0x7f2a7006ea40 addr v1:172.21.15.83:6790/0 2024-03-25T19:27:07.447 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.446+0000 7f2a77460640 10 monclient(hunting): picked mon.noname-a con 0x7f2a70071800 addr v1:172.21.15.83:6789/0 2024-03-25T19:27:07.447 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.446+0000 7f2a77460640 10 monclient(hunting): picked mon.noname-b con 0x7f2a70071c40 addr v1:172.21.15.103:6789/0 2024-03-25T19:27:07.447 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.446+0000 7f2a77460640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a7011b650 con 0x7f2a70071800 2024-03-25T19:27:07.447 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.446+0000 7f2a77460640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a701d37c0 con 0x7f2a70071c40 2024-03-25T19:27:07.447 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.446+0000 7f2a77460640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a7011d2a0 con 0x7f2a7006ea40 2024-03-25T19:27:07.447 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.446+0000 7f2a77460640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:07.448 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.446+0000 7f2a77460640 10 monclient(hunting): authenticate will time out at 6484.717921s 2024-03-25T19:27:07.449 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.449+0000 7f2a759fd640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f2a70071c40 0x7f2a701ddaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:56826/0 (socket says 172.21.15.83:56826) 2024-03-25T19:27:07.449 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.449+0000 7f2a759fd640 1 -- 172.21.15.83:0/2319613355 learned_addr learned my addr 172.21.15.83:0/2319613355 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:27:07.456 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.457+0000 7f2a65ffb640 10 client.?.objecter ms_handle_connect 0x7f2a70071c40 2024-03-25T19:27:07.456 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.457+0000 7f2a65ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:07.468 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.468+0000 7f2a65ffb640 1 -- 172.21.15.83:0/2319613355 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3835240076 0 0) 0x7f2a701d37c0 con 0x7f2a70071c40 2024-03-25T19:27:07.468 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.468+0000 7f2a65ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:27:07.468 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.468+0000 7f2a65ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:07.469 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.468+0000 7f2a65ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:07.469 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.468+0000 7f2a65ffb640 10 monclient(hunting): my global_id is 4567 2024-03-25T19:27:07.469 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.468+0000 7f2a65ffb640 1 -- 172.21.15.83:0/2319613355 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a54004eb0 con 0x7f2a70071c40 2024-03-25T19:27:07.469 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.470+0000 7f2a65ffb640 10 client.?.objecter ms_handle_connect 0x7f2a70071800 2024-03-25T19:27:07.469 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.470+0000 7f2a65ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:07.470 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.470+0000 7f2a65ffb640 10 client.?.objecter ms_handle_connect 0x7f2a7006ea40 2024-03-25T19:27:07.470 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.470+0000 7f2a65ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:07.479 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.479+0000 7f2a65ffb640 1 -- 172.21.15.83:0/2319613355 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 668517958 0 0) 0x7f2a54004eb0 con 0x7f2a70071c40 2024-03-25T19:27:07.479 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.479+0000 7f2a65ffb640 1 -- 172.21.15.83:0/2319613355 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a701d37c0 con 0x7f2a70071c40 2024-03-25T19:27:07.479 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.479+0000 7f2a65ffb640 1 -- 172.21.15.83:0/2319613355 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f2a68001b30 con 0x7f2a70071c40 2024-03-25T19:27:07.479 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.479+0000 7f2a65ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:07.479 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.479+0000 7f2a65ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:27:07.479 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.479+0000 7f2a65ffb640 10 monclient(hunting): dump: 2024-03-25T19:27:07.479 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:27:07.479 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:27:07.480 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:07.480 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:07.480 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:07.480 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:27:07.480 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:27:07.480 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:27:07.480 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:27:07.480 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:27:07.480 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.479+0000 7f2a65ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:27:07.488 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.488+0000 7f2a65ffb640 1 -- 172.21.15.83:0/2319613355 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3611892861 0 0) 0x7f2a7011b650 con 0x7f2a70071800 2024-03-25T19:27:07.488 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.488+0000 7f2a65ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:27:07.488 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.488+0000 7f2a65ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:07.488 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.488+0000 7f2a65ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:07.488 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.488+0000 7f2a65ffb640 10 monclient(hunting): my global_id is 4557 2024-03-25T19:27:07.489 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.488+0000 7f2a65ffb640 1 -- 172.21.15.83:0/2319613355 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a54004eb0 con 0x7f2a70071800 2024-03-25T19:27:07.489 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.490+0000 7f2a65ffb640 1 -- 172.21.15.83:0/2319613355 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1557571139 0 0) 0x7f2a7011d2a0 con 0x7f2a7006ea40 2024-03-25T19:27:07.489 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.490+0000 7f2a65ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:27:07.489 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.490+0000 7f2a65ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:07.489 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.490+0000 7f2a65ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:07.490 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.490+0000 7f2a65ffb640 10 monclient(hunting): my global_id is 4574 2024-03-25T19:27:07.490 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.490+0000 7f2a65ffb640 1 -- 172.21.15.83:0/2319613355 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a7011b650 con 0x7f2a7006ea40 2024-03-25T19:27:07.503 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.503+0000 7f2a65ffb640 1 -- 172.21.15.83:0/2319613355 <== mon.1 v1:172.21.15.103:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1939781453 0 0) 0x7f2a701d37c0 con 0x7f2a70071c40 2024-03-25T19:27:07.503 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.503+0000 7f2a65ffb640 1 -- 172.21.15.83:0/2319613355 >> v1:172.21.15.83:6790/0 conn(0x7f2a7006ea40 legacy=0x7f2a700ad240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:07.503 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.503+0000 7f2a65ffb640 1 -- 172.21.15.83:0/2319613355 >> v1:172.21.15.83:6789/0 conn(0x7f2a70071800 legacy=0x7f2a701d30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:07.503 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.503+0000 7f2a65ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:27:07.504 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.503+0000 7f2a65ffb640 1 monclient: found mon.b 2024-03-25T19:27:07.504 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.503+0000 7f2a65ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:07.504 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.503+0000 7f2a65ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:27:07.504 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.503+0000 7f2a65ffb640 1 -- 172.21.15.83:0/2319613355 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a7011b8a0 con 0x7f2a70071c40 2024-03-25T19:27:07.504 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.503+0000 7f2a65ffb640 10 monclient: _finish_auth 0 2024-03-25T19:27:07.504 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.503+0000 7f2a65ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:27:07.504 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.503+0000 7f2a65ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:07.504 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.503+0000 7f2a77460640 5 monclient: authenticate success, global_id 4567 2024-03-25T19:27:07.504 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.503+0000 7f2a77460640 10 monclient: _renew_subs 2024-03-25T19:27:07.504 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.503+0000 7f2a77460640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:27:07.504 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.503+0000 7f2a77460640 1 -- 172.21.15.83:0/2319613355 --> v1:172.21.15.103:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a7013da60 con 0x7f2a70071c40 2024-03-25T19:27:07.504 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.503+0000 7f2a77460640 10 client.4567.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:27:07.504 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.503+0000 7f2a77460640 10 monclient: _renew_subs 2024-03-25T19:27:07.504 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.503+0000 7f2a77460640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:27:07.504 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.503+0000 7f2a77460640 1 -- 172.21.15.83:0/2319613355 --> v1:172.21.15.103:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a7011bf80 con 0x7f2a70071c40 2024-03-25T19:27:07.505 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.503+0000 7f2a77460640 1 librados: init done 2024-03-25T19:27:07.505 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.504+0000 7f2a77460640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:07.505 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.504+0000 7f2a77460640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:07.505 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.504+0000 7f2a77460640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:27:07.505 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.504+0000 7f2a77460640 1 -- 172.21.15.83:0/2319613355 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a7013dd60 con 0x7f2a70071c40 2024-03-25T19:27:07.793 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.793+0000 7f2a65ffb640 1 -- 172.21.15.83:0/2319613355 <== mon.1 v1:172.21.15.103:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a68002020 con 0x7f2a70071c40 2024-03-25T19:27:07.793 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.793+0000 7f2a65ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:07.793 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.793+0000 7f2a65ffb640 1 -- 172.21.15.83:0/2319613355 <== mon.1 v1:172.21.15.103:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f2a68004d00 con 0x7f2a70071c40 2024-03-25T19:27:07.793 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.793+0000 7f2a65ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:07.793 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.793+0000 7f2a65ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:27:07.793 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.793+0000 7f2a65ffb640 10 monclient: dump: 2024-03-25T19:27:07.793 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:27:07.794 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:27:07.794 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:07.794 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:07.794 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:07.794 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:27:07.794 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:27:07.794 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:27:07.794 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:27:07.794 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:27:07.794 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.793+0000 7f2a677fe640 4 set_mon_vals no callback set 2024-03-25T19:27:07.795 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.794+0000 7f2a65ffb640 1 -- 172.21.15.83:0/2319613355 <== mon.1 v1:172.21.15.103:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7f2a68004f20 con 0x7f2a70071c40 2024-03-25T19:27:07.795 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.795+0000 7f2a751fc640 10 monclient: get_auth_request con 0x7f2a54070020 auth_method 0 2024-03-25T19:27:07.795 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.795+0000 7f2a65ffb640 10 client.4567.objecter ms_handle_connect 0x7f2a54070020 2024-03-25T19:27:07.799 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.799+0000 7f2a65ffb640 1 -- 172.21.15.83:0/2319613355 <== mon.1 v1:172.21.15.103:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7f2a68054b70 con 0x7f2a70071c40 2024-03-25T19:27:07.799 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.799+0000 7f2a65ffb640 10 client.4567.objecter ms_dispatch 0x7f2a70000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:27:07.799 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.799+0000 7f2a65ffb640 3 client.4567.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:27:07.799 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.799+0000 7f2a65ffb640 3 client.4567.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:27:07.799 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.799+0000 7f2a65ffb640 20 client.4567.objecter dump_active .. 0 homeless 2024-03-25T19:27:07.800 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.800+0000 7f2a65ffb640 1 -- 172.21.15.83:0/2319613355 <== mon.1 v1:172.21.15.103:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2a680504b0 con 0x7f2a70071c40 2024-03-25T19:27:07.800 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.800+0000 7f2a65ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:07.800 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:07.800+0000 7f2a65ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:27:08.054 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:08.054+0000 7f2a77460640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:08.054 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:08.054+0000 7f2a77460640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:08.055 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:08.054+0000 7f2a77460640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:27:08.055 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:08.054+0000 7f2a77460640 1 -- 172.21.15.83:0/2319613355 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2a70074d80 con 0x7f2a70071c40 2024-03-25T19:27:08.082 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:08.082+0000 7f2a65ffb640 1 -- 172.21.15.83:0/2319613355 <== mon.1 v1:172.21.15.103:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7f2a70074d80 con 0x7f2a70071c40 2024-03-25T19:27:08.082 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:08.082+0000 7f2a65ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:08.082 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:08.082+0000 7f2a65ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:27:08.082 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:27:08.082 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:27:08.085 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:08.084+0000 7f2a77460640 10 librados: watch_flush enter 2024-03-25T19:27:08.085 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:08.084+0000 7f2a77460640 10 librados: watch_flush exit 2024-03-25T19:27:08.085 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:08.084+0000 7f2a77460640 20 client.4567.objecter shutdown clearing up homeless session... 2024-03-25T19:27:08.085 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:08.084+0000 7f2a77460640 10 client.4567.objecter successfully canceled tick 2024-03-25T19:27:08.085 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:08.085+0000 7f2a77460640 1 -- 172.21.15.83:0/2319613355 >> v1:172.21.15.103:6800/345072327 conn(0x7f2a54070020 legacy=0x7f2a540724e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:08.085 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:08.085+0000 7f2a77460640 10 monclient: shutdown 2024-03-25T19:27:08.085 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:08.085+0000 7f2a77460640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:08.085 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:08.085+0000 7f2a77460640 1 -- 172.21.15.83:0/2319613355 >> v1:172.21.15.103:6789/0 conn(0x7f2a70071c40 legacy=0x7f2a701ddaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:08.085 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:08.085+0000 7f2a77460640 1 -- 172.21.15.83:0/2319613355 shutdown_connections 2024-03-25T19:27:08.085 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:08.085+0000 7f2a77460640 1 -- 172.21.15.83:0/2319613355 >> 172.21.15.83:0/2319613355 conn(0x7f2a7006b630 msgr2=0x7f2a700ae850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:08.085 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:08.085+0000 7f2a77460640 1 -- 172.21.15.83:0/2319613355 shutdown_connections 2024-03-25T19:27:08.085 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:08.085+0000 7f2a77460640 1 -- 172.21.15.83:0/2319613355 wait complete. 2024-03-25T19:27:08.086 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:08.085+0000 7f2a77460640 1 librados: shutdown 2024-03-25T19:27:08.097 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:27:08.465 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:07:50.392 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:08.484 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:07:50.411 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:08.514 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:07:50.424 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:08.518 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:07:50.446 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:08.529 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:07:50.439 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:08.530 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:07:50.458 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:08.533 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:07:50.461 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:08.548 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:07:50.458 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:08.573 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:07:50.501 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:14.904 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:27:15.021 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.021+0000 7f0420332640 10 monclient: get_monmap_and_config 2024-03-25T19:27:15.021 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.021+0000 7f0420332640 10 monclient: build_initial_monmap 2024-03-25T19:27:15.021 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.021+0000 7f0420332640 10 monclient: monmap: 2024-03-25T19:27:15.021 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:27:15.021 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:27:15.021 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:27:15.022170+0000 2024-03-25T19:27:15.021 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:27:15.022170+0000 2024-03-25T19:27:15.021 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:15.021 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:27:15.021 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:27:15.021 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:27:15.021 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:27:15.021 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:27:15.023 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.021+0000 7f0420332640 1 Processor -- start 2024-03-25T19:27:15.023 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.022+0000 7f0420332640 1 -- start start 2024-03-25T19:27:15.023 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.022+0000 7f0420332640 10 monclient: init 2024-03-25T19:27:15.023 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.022+0000 7f0420332640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:15.023 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.022+0000 7f0420332640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:27:15.023 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.022+0000 7f0420332640 10 monclient(hunting): picked mon.noname-c con 0x7f04180a98f0 addr v1:172.21.15.103:6789/0 2024-03-25T19:27:15.023 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.022+0000 7f0420332640 10 monclient(hunting): picked mon.noname-b con 0x7f041813f710 addr v1:172.21.15.83:6790/0 2024-03-25T19:27:15.023 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.022+0000 7f0420332640 10 monclient(hunting): picked mon.noname-a con 0x7f04181430c0 addr v1:172.21.15.83:6789/0 2024-03-25T19:27:15.023 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.022+0000 7f0420332640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0418142520 con 0x7f04181430c0 2024-03-25T19:27:15.023 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.023+0000 7f0420332640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04180b28a0 con 0x7f04180a98f0 2024-03-25T19:27:15.023 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.023+0000 7f0420332640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04180b2a30 con 0x7f041813f710 2024-03-25T19:27:15.023 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.023+0000 7f0420332640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:15.023 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.023+0000 7f0420332640 10 monclient(hunting): authenticate will time out at 6492.294767s 2024-03-25T19:27:15.029 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.030+0000 7f041e0ce640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f04180a98f0 0x7f04180a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:33300/0 (socket says 172.21.15.83:33300) 2024-03-25T19:27:15.029 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.030+0000 7f041e0ce640 1 -- 172.21.15.83:0/1857551312 learned_addr learned my addr 172.21.15.83:0/1857551312 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:27:15.041 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.041+0000 7f041d0cc640 1 -- 172.21.15.83:0/1857551312 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3998975173 0 0) 0x7f04180b28a0 con 0x7f04180a98f0 2024-03-25T19:27:15.041 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.041+0000 7f041d0cc640 10 monclient(hunting): _negotiate 2024-03-25T19:27:15.041 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.041+0000 7f041d0cc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:15.041 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.041+0000 7f041d0cc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:15.041 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.041+0000 7f041d0cc640 10 monclient(hunting): my global_id is 4570 2024-03-25T19:27:15.041 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.041+0000 7f041d0cc640 1 -- 172.21.15.83:0/1857551312 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0404003520 con 0x7f04180a98f0 2024-03-25T19:27:15.053 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.053+0000 7f041d0cc640 1 -- 172.21.15.83:0/1857551312 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 898589853 0 0) 0x7f0418142520 con 0x7f04181430c0 2024-03-25T19:27:15.053 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.053+0000 7f041d0cc640 10 monclient(hunting): _negotiate 2024-03-25T19:27:15.053 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.053+0000 7f041d0cc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:15.053 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.053+0000 7f041d0cc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:15.053 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.053+0000 7f041d0cc640 10 monclient(hunting): my global_id is 4560 2024-03-25T19:27:15.053 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.053+0000 7f041d0cc640 1 -- 172.21.15.83:0/1857551312 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04180b28a0 con 0x7f04181430c0 2024-03-25T19:27:15.056 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.056+0000 7f041d0cc640 1 -- 172.21.15.83:0/1857551312 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2356819492 0 0) 0x7f0404003520 con 0x7f04180a98f0 2024-03-25T19:27:15.056 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.056+0000 7f041d0cc640 1 -- 172.21.15.83:0/1857551312 >> v1:172.21.15.83:6790/0 conn(0x7f041813f710 legacy=0x7f04181429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:15.056 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.056+0000 7f041d0cc640 1 -- 172.21.15.83:0/1857551312 >> v1:172.21.15.83:6789/0 conn(0x7f04181430c0 legacy=0x7f041814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:15.056 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.056+0000 7f041d0cc640 10 monclient: _finish_hunting 0 2024-03-25T19:27:15.056 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.056+0000 7f041d0cc640 1 monclient: found mon.noname-c 2024-03-25T19:27:15.056 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.056+0000 7f041d0cc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:15.056 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.056+0000 7f041d0cc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.103:6789/0 2024-03-25T19:27:15.056 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.056+0000 7f041d0cc640 1 -- 172.21.15.83:0/1857551312 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04180b44c0 con 0x7f04180a98f0 2024-03-25T19:27:15.056 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.056+0000 7f041d0cc640 10 monclient: _finish_auth 0 2024-03-25T19:27:15.056 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.056+0000 7f041d0cc640 10 monclient: _check_auth_tickets 2024-03-25T19:27:15.056 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.056+0000 7f041d0cc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:15.056 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.056+0000 7f041d0cc640 1 -- 172.21.15.83:0/1857551312 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f0414001690 con 0x7f04180a98f0 2024-03-25T19:27:15.056 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.056+0000 7f041d0cc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:15.056 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.056+0000 7f041d0cc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:27:15.057 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.056+0000 7f041d0cc640 10 monclient: dump: 2024-03-25T19:27:15.057 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:27:15.057 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:27:15.057 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:15.057 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:15.057 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:15.057 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:27:15.057 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:27:15.057 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:27:15.057 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:27:15.057 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:27:15.057 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.056+0000 7f0420332640 5 monclient: authenticate success, global_id 4570 2024-03-25T19:27:15.057 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.056+0000 7f0420332640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:27:15.070 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.071+0000 7f041d0cc640 1 -- 172.21.15.83:0/1857551312 <== mon.1 v1:172.21.15.103:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04140017f0 con 0x7f04180a98f0 2024-03-25T19:27:15.070 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.071+0000 7f041d0cc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:15.070 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.071+0000 7f041d0cc640 1 -- 172.21.15.83:0/1857551312 <== mon.1 v1:172.21.15.103:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f0414001c00 con 0x7f04180a98f0 2024-03-25T19:27:15.071 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.071+0000 7f041d0cc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:15.071 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.071+0000 7f041d0cc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:27:15.071 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.071+0000 7f041d0cc640 10 monclient: dump: 2024-03-25T19:27:15.071 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:27:15.071 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:27:15.071 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:15.071 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:15.071 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:15.071 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:27:15.071 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:27:15.071 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:27:15.071 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:27:15.071 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:27:15.071 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.071+0000 7f0420332640 10 monclient: get_monmap_and_config success 2024-03-25T19:27:15.071 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.071+0000 7f0420332640 4 set_mon_vals no callback set 2024-03-25T19:27:15.074 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.072+0000 7f0420332640 10 monclient: shutdown 2024-03-25T19:27:15.074 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.072+0000 7f0420332640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:15.074 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.072+0000 7f0420332640 1 -- 172.21.15.83:0/1857551312 >> v1:172.21.15.103:6789/0 conn(0x7f04180a98f0 legacy=0x7f04180a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:15.074 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.072+0000 7f0420332640 1 -- 172.21.15.83:0/1857551312 shutdown_connections 2024-03-25T19:27:15.074 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.072+0000 7f0420332640 1 -- 172.21.15.83:0/1857551312 >> 172.21.15.83:0/1857551312 conn(0x7f04180b7ea0 msgr2=0x7f04180b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:15.074 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.072+0000 7f0420332640 1 -- 172.21.15.83:0/1857551312 shutdown_connections 2024-03-25T19:27:15.074 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.072+0000 7f0420332640 1 -- 172.21.15.83:0/1857551312 wait complete. 2024-03-25T19:27:15.074 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.072+0000 7f0420332640 10 monclient: build_initial_monmap 2024-03-25T19:27:15.074 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.073+0000 7f0420332640 10 monclient: monmap: 2024-03-25T19:27:15.074 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:27:15.074 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:27:15.075 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:27:15.075 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:27:15.075 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:15.075 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:27:15.075 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:27:15.075 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:27:15.075 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:27:15.075 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:27:15.075 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.073+0000 7f0420332640 1 librados: starting msgr at 2024-03-25T19:27:15.075 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.073+0000 7f0420332640 1 librados: starting objecter 2024-03-25T19:27:15.075 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.073+0000 7f0420332640 1 Processor -- start 2024-03-25T19:27:15.075 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.073+0000 7f0420332640 1 -- start start 2024-03-25T19:27:15.075 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.073+0000 7f0420332640 1 librados: setting wanted keys 2024-03-25T19:27:15.075 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.073+0000 7f0420332640 1 librados: calling monclient init 2024-03-25T19:27:15.075 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.073+0000 7f0420332640 10 monclient: init 2024-03-25T19:27:15.075 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.073+0000 7f0420332640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:15.076 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.073+0000 7f0420332640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:27:15.076 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.073+0000 7f0420332640 10 monclient(hunting): picked mon.noname-c con 0x7f04180a98f0 addr v1:172.21.15.83:6790/0 2024-03-25T19:27:15.076 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.074+0000 7f0420332640 10 monclient(hunting): picked mon.noname-a con 0x7f041813f710 addr v1:172.21.15.83:6789/0 2024-03-25T19:27:15.076 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.074+0000 7f0420332640 10 monclient(hunting): picked mon.noname-b con 0x7f04181430c0 addr v1:172.21.15.103:6789/0 2024-03-25T19:27:15.076 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.074+0000 7f0420332640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0418138250 con 0x7f041813f710 2024-03-25T19:27:15.076 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.074+0000 7f0420332640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0418137d50 con 0x7f04181430c0 2024-03-25T19:27:15.076 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.074+0000 7f0420332640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0418137ee0 con 0x7f04180a98f0 2024-03-25T19:27:15.076 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.074+0000 7f0420332640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:15.076 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.074+0000 7f0420332640 10 monclient(hunting): authenticate will time out at 6492.345884s 2024-03-25T19:27:15.079 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.080+0000 7f041e0ce640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f04180a98f0 0x7f0418137640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:48658/0 (socket says 172.21.15.83:48658) 2024-03-25T19:27:15.079 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.080+0000 7f041e0ce640 1 -- 172.21.15.83:0/2236633146 learned_addr learned my addr 172.21.15.83:0/2236633146 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:27:15.081 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.082+0000 7f040effd640 10 client.?.objecter ms_handle_connect 0x7f04180a98f0 2024-03-25T19:27:15.081 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.082+0000 7f040effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:15.082 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.083+0000 7f040effd640 10 client.?.objecter ms_handle_connect 0x7f041813f710 2024-03-25T19:27:15.083 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.083+0000 7f040effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:15.089 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.088+0000 7f040effd640 1 -- 172.21.15.83:0/2236633146 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2622511154 0 0) 0x7f0418137ee0 con 0x7f04180a98f0 2024-03-25T19:27:15.089 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.088+0000 7f040effd640 10 monclient(hunting): _negotiate 2024-03-25T19:27:15.089 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.088+0000 7f040effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:15.090 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.088+0000 7f040effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:15.090 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.088+0000 7f040effd640 10 monclient(hunting): my global_id is 4580 2024-03-25T19:27:15.090 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.088+0000 7f040effd640 1 -- 172.21.15.83:0/2236633146 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03fc004eb0 con 0x7f04180a98f0 2024-03-25T19:27:15.090 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.089+0000 7f040effd640 10 client.?.objecter ms_handle_connect 0x7f04181430c0 2024-03-25T19:27:15.090 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.089+0000 7f040effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:15.094 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.094+0000 7f040effd640 1 -- 172.21.15.83:0/2236633146 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 357292633 0 0) 0x7f0418138250 con 0x7f041813f710 2024-03-25T19:27:15.094 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.094+0000 7f040effd640 10 monclient(hunting): _negotiate 2024-03-25T19:27:15.094 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.094+0000 7f040effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:15.094 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.094+0000 7f040effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:15.094 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.094+0000 7f040effd640 10 monclient(hunting): my global_id is 4563 2024-03-25T19:27:15.094 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.094+0000 7f040effd640 1 -- 172.21.15.83:0/2236633146 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0418137ee0 con 0x7f041813f710 2024-03-25T19:27:15.105 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.105+0000 7f040effd640 1 -- 172.21.15.83:0/2236633146 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 545915126 0 0) 0x7f03fc004eb0 con 0x7f04180a98f0 2024-03-25T19:27:15.105 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.105+0000 7f040effd640 1 -- 172.21.15.83:0/2236633146 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0418138250 con 0x7f04180a98f0 2024-03-25T19:27:15.105 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.105+0000 7f040effd640 1 -- 172.21.15.83:0/2236633146 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f0414001bc0 con 0x7f04180a98f0 2024-03-25T19:27:15.105 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.105+0000 7f040effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:15.105 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.105+0000 7f040effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:27:15.105 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.105+0000 7f040effd640 10 monclient(hunting): dump: 2024-03-25T19:27:15.105 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:27:15.105 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:27:15.105 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:15.105 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:15.105 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:15.105 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:27:15.106 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:27:15.106 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:27:15.106 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:27:15.106 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:27:15.106 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.105+0000 7f040effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:27:15.112 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.112+0000 7f040effd640 1 -- 172.21.15.83:0/2236633146 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2480490477 0 0) 0x7f0418137d50 con 0x7f04181430c0 2024-03-25T19:27:15.112 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.112+0000 7f040effd640 10 monclient(hunting): _negotiate 2024-03-25T19:27:15.112 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.112+0000 7f040effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:15.112 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.112+0000 7f040effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:15.112 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.112+0000 7f040effd640 10 monclient(hunting): my global_id is 4573 2024-03-25T19:27:15.112 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.112+0000 7f040effd640 1 -- 172.21.15.83:0/2236633146 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03fc004eb0 con 0x7f04181430c0 2024-03-25T19:27:15.120 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.120+0000 7f040effd640 1 -- 172.21.15.83:0/2236633146 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 543643561 0 0) 0x7f0418138250 con 0x7f04180a98f0 2024-03-25T19:27:15.120 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.120+0000 7f040effd640 1 -- 172.21.15.83:0/2236633146 >> v1:172.21.15.103:6789/0 conn(0x7f04181430c0 legacy=0x7f04181d7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:15.120 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.120+0000 7f040effd640 1 -- 172.21.15.83:0/2236633146 >> v1:172.21.15.83:6789/0 conn(0x7f041813f710 legacy=0x7f04181d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:15.120 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.120+0000 7f040effd640 10 monclient: _finish_hunting 0 2024-03-25T19:27:15.120 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.120+0000 7f040effd640 1 monclient: found mon.c 2024-03-25T19:27:15.120 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.120+0000 7f040effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:15.120 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.120+0000 7f040effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:27:15.120 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.120+0000 7f040effd640 1 -- 172.21.15.83:0/2236633146 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04180b7730 con 0x7f04180a98f0 2024-03-25T19:27:15.120 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.120+0000 7f040effd640 10 monclient: _finish_auth 0 2024-03-25T19:27:15.120 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.120+0000 7f040effd640 10 monclient: _check_auth_tickets 2024-03-25T19:27:15.120 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.120+0000 7f040effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:15.120 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.120+0000 7f0420332640 5 monclient: authenticate success, global_id 4580 2024-03-25T19:27:15.121 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.120+0000 7f0420332640 10 monclient: _renew_subs 2024-03-25T19:27:15.121 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.120+0000 7f0420332640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:27:15.121 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.120+0000 7f0420332640 1 -- 172.21.15.83:0/2236633146 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04180b7920 con 0x7f04180a98f0 2024-03-25T19:27:15.121 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.120+0000 7f0420332640 10 client.4580.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:27:15.121 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.120+0000 7f0420332640 10 monclient: _renew_subs 2024-03-25T19:27:15.121 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.120+0000 7f0420332640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:27:15.121 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.120+0000 7f0420332640 1 -- 172.21.15.83:0/2236633146 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f04181e33d0 con 0x7f04180a98f0 2024-03-25T19:27:15.121 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.120+0000 7f0420332640 1 librados: init done 2024-03-25T19:27:15.121 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.121+0000 7f0420332640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:15.122 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.121+0000 7f0420332640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:15.122 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.121+0000 7f0420332640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:27:15.122 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.121+0000 7f0420332640 1 -- 172.21.15.83:0/2236633146 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04180627b0 con 0x7f04180a98f0 2024-03-25T19:27:15.409 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.409+0000 7f040effd640 1 -- 172.21.15.83:0/2236633146 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04140011f0 con 0x7f04180a98f0 2024-03-25T19:27:15.409 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.409+0000 7f040effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:15.409 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.409+0000 7f040effd640 1 -- 172.21.15.83:0/2236633146 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f0414004c30 con 0x7f04180a98f0 2024-03-25T19:27:15.409 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.409+0000 7f040effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:15.409 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.409+0000 7f040effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:27:15.409 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.409+0000 7f040effd640 10 monclient: dump: 2024-03-25T19:27:15.409 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:27:15.409 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:27:15.409 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:15.409 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:15.409 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:15.409 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:27:15.410 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:27:15.410 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:27:15.410 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:27:15.410 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:27:15.410 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.409+0000 7f041c8cb640 4 set_mon_vals no callback set 2024-03-25T19:27:15.410 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.410+0000 7f040effd640 1 -- 172.21.15.83:0/2236633146 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7f041401ac80 con 0x7f04180a98f0 2024-03-25T19:27:15.411 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.410+0000 7f041d8cd640 10 monclient: get_auth_request con 0x7f03fc06ddb0 auth_method 0 2024-03-25T19:27:15.411 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.411+0000 7f040effd640 10 client.4580.objecter ms_handle_connect 0x7f03fc06ddb0 2024-03-25T19:27:15.415 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.415+0000 7f040effd640 1 -- 172.21.15.83:0/2236633146 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7f04140548d0 con 0x7f04180a98f0 2024-03-25T19:27:15.416 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.416+0000 7f040effd640 10 client.4580.objecter ms_dispatch 0x7f0418000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:27:15.416 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.416+0000 7f040effd640 3 client.4580.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:27:15.416 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.416+0000 7f040effd640 3 client.4580.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:27:15.416 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.416+0000 7f040effd640 20 client.4580.objecter dump_active .. 0 homeless 2024-03-25T19:27:15.416 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.416+0000 7f040effd640 1 -- 172.21.15.83:0/2236633146 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0414050210 con 0x7f04180a98f0 2024-03-25T19:27:15.416 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.416+0000 7f040effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:15.416 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.416+0000 7f040effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:27:15.639 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.639+0000 7f0420332640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:15.639 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.639+0000 7f0420332640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:15.639 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.639+0000 7f0420332640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:27:15.639 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.639+0000 7f0420332640 1 -- 172.21.15.83:0/2236633146 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f041813df50 con 0x7f04180a98f0 2024-03-25T19:27:15.677 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.677+0000 7f040effd640 1 -- 172.21.15.83:0/2236633146 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7f0414052310 con 0x7f04180a98f0 2024-03-25T19:27:15.677 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.677+0000 7f040effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:15.677 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.677+0000 7f040effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:27:15.677 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:27:15.678 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:27:15.680 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.680+0000 7f0420332640 10 librados: watch_flush enter 2024-03-25T19:27:15.680 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.680+0000 7f0420332640 10 librados: watch_flush exit 2024-03-25T19:27:15.680 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.680+0000 7f0420332640 20 client.4580.objecter shutdown clearing up homeless session... 2024-03-25T19:27:15.680 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.680+0000 7f0420332640 10 client.4580.objecter successfully canceled tick 2024-03-25T19:27:15.680 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.680+0000 7f0420332640 1 -- 172.21.15.83:0/2236633146 >> v1:172.21.15.103:6800/345072327 conn(0x7f03fc06ddb0 legacy=0x7f03fc070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:15.680 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.680+0000 7f0420332640 10 monclient: shutdown 2024-03-25T19:27:15.680 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.680+0000 7f0420332640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:15.680 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.680+0000 7f0420332640 1 -- 172.21.15.83:0/2236633146 >> v1:172.21.15.83:6790/0 conn(0x7f04180a98f0 legacy=0x7f0418137640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:15.680 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.680+0000 7f0420332640 1 -- 172.21.15.83:0/2236633146 shutdown_connections 2024-03-25T19:27:15.680 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.680+0000 7f0420332640 1 -- 172.21.15.83:0/2236633146 >> 172.21.15.83:0/2236633146 conn(0x7f04180b7ea0 msgr2=0x7f0418074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:15.680 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.680+0000 7f0420332640 1 -- 172.21.15.83:0/2236633146 shutdown_connections 2024-03-25T19:27:15.681 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.680+0000 7f0420332640 1 -- 172.21.15.83:0/2236633146 wait complete. 2024-03-25T19:27:15.681 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:15.680+0000 7f0420332640 1 librados: shutdown 2024-03-25T19:27:15.691 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:27:22.497 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:27:22.614 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.614+0000 7f7a15a6b640 10 monclient: get_monmap_and_config 2024-03-25T19:27:22.614 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.614+0000 7f7a15a6b640 10 monclient: build_initial_monmap 2024-03-25T19:27:22.614 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.614+0000 7f7a15a6b640 10 monclient: monmap: 2024-03-25T19:27:22.615 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:27:22.615 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:27:22.615 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:27:22.615733+0000 2024-03-25T19:27:22.615 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:27:22.615733+0000 2024-03-25T19:27:22.615 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:22.615 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:27:22.615 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:27:22.615 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:27:22.615 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:27:22.615 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:27:22.616 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.615+0000 7f7a15a6b640 1 Processor -- start 2024-03-25T19:27:22.616 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.615+0000 7f7a15a6b640 1 -- start start 2024-03-25T19:27:22.617 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.615+0000 7f7a15a6b640 10 monclient: init 2024-03-25T19:27:22.617 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.616+0000 7f7a15a6b640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:22.617 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.616+0000 7f7a15a6b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:27:22.617 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.616+0000 7f7a15a6b640 10 monclient(hunting): picked mon.noname-a con 0x7f7a100b4d60 addr v1:172.21.15.83:6789/0 2024-03-25T19:27:22.617 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.616+0000 7f7a15a6b640 10 monclient(hunting): picked mon.noname-c con 0x7f7a100b1730 addr v1:172.21.15.103:6789/0 2024-03-25T19:27:22.617 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.616+0000 7f7a15a6b640 10 monclient(hunting): picked mon.noname-b con 0x7f7a1014be90 addr v1:172.21.15.83:6790/0 2024-03-25T19:27:22.617 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.616+0000 7f7a15a6b640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a100af940 con 0x7f7a100b4d60 2024-03-25T19:27:22.617 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.616+0000 7f7a15a6b640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a100b0a70 con 0x7f7a100b1730 2024-03-25T19:27:22.617 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.616+0000 7f7a15a6b640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a100b0d50 con 0x7f7a1014be90 2024-03-25T19:27:22.617 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.616+0000 7f7a15a6b640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:22.617 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.616+0000 7f7a15a6b640 10 monclient(hunting): authenticate will time out at 6499.888309s 2024-03-25T19:27:22.621 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.622+0000 7f7a0effd640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f7a100b1730 0x7f7a100b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:57642/0 (socket says 172.21.15.83:57642) 2024-03-25T19:27:22.621 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.622+0000 7f7a0effd640 1 -- 172.21.15.83:0/1995999448 learned_addr learned my addr 172.21.15.83:0/1995999448 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:27:22.631 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.631+0000 7f7a0e7fc640 1 -- 172.21.15.83:0/1995999448 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 768681963 0 0) 0x7f7a100b0a70 con 0x7f7a100b1730 2024-03-25T19:27:22.631 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.631+0000 7f7a0e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:27:22.631 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.631+0000 7f7a0e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:22.631 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.631+0000 7f7a0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:22.631 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.631+0000 7f7a0e7fc640 10 monclient(hunting): my global_id is 4576 2024-03-25T19:27:22.632 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.631+0000 7f7a0e7fc640 1 -- 172.21.15.83:0/1995999448 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79f8003520 con 0x7f7a100b1730 2024-03-25T19:27:22.644 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.644+0000 7f7a0e7fc640 1 -- 172.21.15.83:0/1995999448 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 638933716 0 0) 0x7f79f8003520 con 0x7f7a100b1730 2024-03-25T19:27:22.644 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.644+0000 7f7a0e7fc640 1 -- 172.21.15.83:0/1995999448 >> v1:172.21.15.83:6790/0 conn(0x7f7a1014be90 legacy=0x7f7a1014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:27:22.644 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.644+0000 7f7a0e7fc640 1 -- 172.21.15.83:0/1995999448 >> v1:172.21.15.83:6789/0 conn(0x7f7a100b4d60 legacy=0x7f7a1013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:22.644 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.644+0000 7f7a0e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:27:22.644 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.644+0000 7f7a0e7fc640 1 monclient: found mon.noname-c 2024-03-25T19:27:22.645 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.644+0000 7f7a0e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:22.645 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.644+0000 7f7a0e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.103:6789/0 2024-03-25T19:27:22.645 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.644+0000 7f7a0e7fc640 1 -- 172.21.15.83:0/1995999448 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a100b1030 con 0x7f7a100b1730 2024-03-25T19:27:22.645 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.644+0000 7f7a0e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:27:22.645 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.644+0000 7f7a0e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:27:22.645 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.644+0000 7f7a0e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:22.645 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.644+0000 7f7a0e7fc640 1 -- 172.21.15.83:0/1995999448 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f7a04001690 con 0x7f7a100b1730 2024-03-25T19:27:22.645 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.644+0000 7f7a15a6b640 5 monclient: authenticate success, global_id 4576 2024-03-25T19:27:22.645 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.644+0000 7f7a15a6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:27:22.645 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.644+0000 7f7a0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:22.645 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.644+0000 7f7a0e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:27:22.645 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.644+0000 7f7a0e7fc640 10 monclient: dump: 2024-03-25T19:27:22.645 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:27:22.645 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:27:22.645 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:22.646 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:22.646 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:22.646 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:27:22.646 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:27:22.646 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:27:22.646 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:27:22.646 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:27:22.646 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.644+0000 7f7a15a6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:27:22.658 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.658+0000 7f7a0e7fc640 1 -- 172.21.15.83:0/1995999448 <== mon.1 v1:172.21.15.103:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a04001860 con 0x7f7a100b1730 2024-03-25T19:27:22.694 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.658+0000 7f7a0e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:22.694 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.658+0000 7f7a0e7fc640 1 -- 172.21.15.83:0/1995999448 <== mon.1 v1:172.21.15.103:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f7a04001cb0 con 0x7f7a100b1730 2024-03-25T19:27:22.694 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.658+0000 7f7a0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:22.694 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.658+0000 7f7a0e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:27:22.694 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.658+0000 7f7a0e7fc640 10 monclient: dump: 2024-03-25T19:27:22.694 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:27:22.694 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:27:22.694 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:22.695 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:22.695 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:22.695 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:27:22.695 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:27:22.695 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:27:22.695 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:27:22.695 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:27:22.695 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.658+0000 7f7a15a6b640 10 monclient: get_monmap_and_config success 2024-03-25T19:27:22.695 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.658+0000 7f7a15a6b640 4 set_mon_vals no callback set 2024-03-25T19:27:22.698 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.659+0000 7f7a15a6b640 10 monclient: shutdown 2024-03-25T19:27:22.698 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.659+0000 7f7a15a6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:22.698 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.659+0000 7f7a15a6b640 1 -- 172.21.15.83:0/1995999448 >> v1:172.21.15.103:6789/0 conn(0x7f7a100b1730 legacy=0x7f7a100b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:22.698 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.659+0000 7f7a15a6b640 1 -- 172.21.15.83:0/1995999448 shutdown_connections 2024-03-25T19:27:22.698 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.659+0000 7f7a15a6b640 1 -- 172.21.15.83:0/1995999448 >> 172.21.15.83:0/1995999448 conn(0x7f7a100617a0 msgr2=0x7f7a10061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:22.698 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.659+0000 7f7a15a6b640 1 -- 172.21.15.83:0/1995999448 shutdown_connections 2024-03-25T19:27:22.698 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.659+0000 7f7a15a6b640 1 -- 172.21.15.83:0/1995999448 wait complete. 2024-03-25T19:27:22.698 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.660+0000 7f7a15a6b640 10 monclient: build_initial_monmap 2024-03-25T19:27:22.698 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.660+0000 7f7a15a6b640 10 monclient: monmap: 2024-03-25T19:27:22.698 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:27:22.698 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:27:22.698 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:27:22.699 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:27:22.699 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:22.699 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:27:22.699 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:27:22.699 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:27:22.699 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:27:22.699 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:27:22.699 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.660+0000 7f7a15a6b640 1 librados: starting msgr at 2024-03-25T19:27:22.699 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.660+0000 7f7a15a6b640 1 librados: starting objecter 2024-03-25T19:27:22.699 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.660+0000 7f7a15a6b640 1 Processor -- start 2024-03-25T19:27:22.699 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.661+0000 7f7a15a6b640 1 -- start start 2024-03-25T19:27:22.699 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.661+0000 7f7a15a6b640 1 librados: setting wanted keys 2024-03-25T19:27:22.699 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.661+0000 7f7a15a6b640 1 librados: calling monclient init 2024-03-25T19:27:22.699 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.661+0000 7f7a15a6b640 10 monclient: init 2024-03-25T19:27:22.699 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.661+0000 7f7a15a6b640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:22.699 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.661+0000 7f7a15a6b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:27:22.700 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.661+0000 7f7a15a6b640 10 monclient(hunting): picked mon.noname-a con 0x7f7a100b1730 addr v1:172.21.15.83:6789/0 2024-03-25T19:27:22.700 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.662+0000 7f7a15a6b640 10 monclient(hunting): picked mon.noname-c con 0x7f7a100b4d60 addr v1:172.21.15.83:6790/0 2024-03-25T19:27:22.700 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.662+0000 7f7a15a6b640 10 monclient(hunting): picked mon.noname-b con 0x7f7a1014be90 addr v1:172.21.15.103:6789/0 2024-03-25T19:27:22.700 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.662+0000 7f7a15a6b640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a10139320 con 0x7f7a100b1730 2024-03-25T19:27:22.700 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.662+0000 7f7a15a6b640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a10139ee0 con 0x7f7a1014be90 2024-03-25T19:27:22.700 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.662+0000 7f7a15a6b640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a1013a150 con 0x7f7a100b4d60 2024-03-25T19:27:22.700 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.662+0000 7f7a15a6b640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:22.700 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.662+0000 7f7a15a6b640 10 monclient(hunting): authenticate will time out at 6499.934059s 2024-03-25T19:27:22.700 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.673+0000 7f7a0f7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f7a100b1730 0x7f7a1013b140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60296/0 (socket says 172.21.15.83:60296) 2024-03-25T19:27:22.700 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.673+0000 7f7a0f7fe640 1 -- 172.21.15.83:0/1445403901 learned_addr learned my addr 172.21.15.83:0/1445403901 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:27:22.700 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.675+0000 7f79f7fff640 10 client.?.objecter ms_handle_connect 0x7f7a100b1730 2024-03-25T19:27:22.700 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.675+0000 7f79f7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:22.700 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.686+0000 7f79f7fff640 10 client.?.objecter ms_handle_connect 0x7f7a100b4d60 2024-03-25T19:27:22.700 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.686+0000 7f79f7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:22.700 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.686+0000 7f79f7fff640 1 -- 172.21.15.83:0/1445403901 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1863929897 0 0) 0x7f7a10139320 con 0x7f7a100b1730 2024-03-25T19:27:22.701 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.686+0000 7f79f7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:27:22.701 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.686+0000 7f79f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:22.701 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.686+0000 7f79f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:22.701 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.686+0000 7f79f7fff640 10 monclient(hunting): my global_id is 4569 2024-03-25T19:27:22.701 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.686+0000 7f79f7fff640 1 -- 172.21.15.83:0/1445403901 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79fc004eb0 con 0x7f7a100b1730 2024-03-25T19:27:22.701 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.690+0000 7f79f7fff640 10 client.?.objecter ms_handle_connect 0x7f7a1014be90 2024-03-25T19:27:22.701 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.690+0000 7f79f7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:22.702 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.700+0000 7f79f7fff640 1 -- 172.21.15.83:0/1445403901 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2559369123 0 0) 0x7f79fc004eb0 con 0x7f7a100b1730 2024-03-25T19:27:22.702 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.700+0000 7f79f7fff640 1 -- 172.21.15.83:0/1445403901 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a10139320 con 0x7f7a100b1730 2024-03-25T19:27:22.702 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.700+0000 7f79f7fff640 1 -- 172.21.15.83:0/1445403901 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f7a00001a30 con 0x7f7a100b1730 2024-03-25T19:27:22.702 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.700+0000 7f79f7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:22.702 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.700+0000 7f79f7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:27:22.702 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.700+0000 7f79f7fff640 10 monclient(hunting): dump: 2024-03-25T19:27:22.702 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:27:22.702 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:27:22.702 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:22.702 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:22.702 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:22.702 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:27:22.702 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:27:22.703 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:27:22.703 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:27:22.703 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:27:22.703 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.700+0000 7f79f7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:27:22.703 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.703+0000 7f79f7fff640 1 -- 172.21.15.83:0/1445403901 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1463931222 0 0) 0x7f7a10139ee0 con 0x7f7a1014be90 2024-03-25T19:27:22.703 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.703+0000 7f79f7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:27:22.703 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.703+0000 7f79f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:22.703 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.703+0000 7f79f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:22.703 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.703+0000 7f79f7fff640 10 monclient(hunting): my global_id is 4579 2024-03-25T19:27:22.703 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.703+0000 7f79f7fff640 1 -- 172.21.15.83:0/1445403901 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79fc004eb0 con 0x7f7a1014be90 2024-03-25T19:27:22.704 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.704+0000 7f79f7fff640 1 -- 172.21.15.83:0/1445403901 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2647084533 0 0) 0x7f7a1013a150 con 0x7f7a100b4d60 2024-03-25T19:27:22.704 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.704+0000 7f79f7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:27:22.704 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.704+0000 7f79f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:22.704 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.704+0000 7f79f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:22.704 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.704+0000 7f79f7fff640 10 monclient(hunting): my global_id is 4583 2024-03-25T19:27:22.704 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.704+0000 7f79f7fff640 1 -- 172.21.15.83:0/1445403901 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a10139ee0 con 0x7f7a100b4d60 2024-03-25T19:27:22.711 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.711+0000 7f79f7fff640 1 -- 172.21.15.83:0/1445403901 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1440046888 0 0) 0x7f7a10139320 con 0x7f7a100b1730 2024-03-25T19:27:22.711 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.711+0000 7f79f7fff640 1 -- 172.21.15.83:0/1445403901 >> v1:172.21.15.83:6790/0 conn(0x7f7a100b4d60 legacy=0x7f7a1013b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:22.712 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.711+0000 7f79f7fff640 1 -- 172.21.15.83:0/1445403901 >> v1:172.21.15.103:6789/0 conn(0x7f7a1014be90 legacy=0x7f7a101e6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:22.712 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.711+0000 7f79f7fff640 10 monclient: _finish_hunting 0 2024-03-25T19:27:22.712 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.711+0000 7f79f7fff640 1 monclient: found mon.a 2024-03-25T19:27:22.712 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.711+0000 7f79f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:22.712 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.711+0000 7f79f7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:27:22.712 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.711+0000 7f79f7fff640 1 -- 172.21.15.83:0/1445403901 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a10142fc0 con 0x7f7a100b1730 2024-03-25T19:27:22.712 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.711+0000 7f79f7fff640 10 monclient: _finish_auth 0 2024-03-25T19:27:22.712 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.711+0000 7f79f7fff640 10 monclient: _check_auth_tickets 2024-03-25T19:27:22.712 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.711+0000 7f79f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:22.712 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.711+0000 7f7a15a6b640 5 monclient: authenticate success, global_id 4569 2024-03-25T19:27:22.712 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.711+0000 7f7a15a6b640 10 monclient: _renew_subs 2024-03-25T19:27:22.712 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.711+0000 7f7a15a6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:27:22.712 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.711+0000 7f7a15a6b640 1 -- 172.21.15.83:0/1445403901 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a10143310 con 0x7f7a100b1730 2024-03-25T19:27:22.712 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.711+0000 7f7a15a6b640 10 client.4569.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:27:22.713 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.711+0000 7f7a15a6b640 10 monclient: _renew_subs 2024-03-25T19:27:22.713 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.711+0000 7f7a15a6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:27:22.713 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.711+0000 7f7a15a6b640 1 -- 172.21.15.83:0/1445403901 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a10142900 con 0x7f7a100b1730 2024-03-25T19:27:22.713 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.711+0000 7f7a15a6b640 1 librados: init done 2024-03-25T19:27:22.713 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.713+0000 7f7a15a6b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:22.713 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.713+0000 7f7a15a6b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:22.713 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.713+0000 7f7a15a6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:27:22.713 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:22.713+0000 7f7a15a6b640 1 -- 172.21.15.83:0/1445403901 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a101390f0 con 0x7f7a100b1730 2024-03-25T19:27:23.030 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:23.031+0000 7f79f7fff640 1 -- 172.21.15.83:0/1445403901 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a00001cb0 con 0x7f7a100b1730 2024-03-25T19:27:23.031 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:23.031+0000 7f79f7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:23.031 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:23.031+0000 7f79f7fff640 1 -- 172.21.15.83:0/1445403901 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f7a000038c0 con 0x7f7a100b1730 2024-03-25T19:27:23.031 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:23.031+0000 7f79f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:23.031 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:23.031+0000 7f79f7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:27:23.031 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:23.031+0000 7f79f7fff640 10 monclient: dump: 2024-03-25T19:27:23.031 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:27:23.031 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:27:23.031 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:23.031 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:23.031 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:23.031 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:27:23.031 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:27:23.031 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:27:23.031 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:27:23.031 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:27:23.032 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:23.031+0000 7f7a0dffb640 4 set_mon_vals no callback set 2024-03-25T19:27:23.032 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:23.032+0000 7f79f7fff640 1 -- 172.21.15.83:0/1445403901 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7f7a00003ae0 con 0x7f7a100b1730 2024-03-25T19:27:23.032 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:23.032+0000 7f7a0effd640 10 monclient: get_auth_request con 0x7f79fc06de80 auth_method 0 2024-03-25T19:27:23.032 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:23.033+0000 7f79f7fff640 10 client.4569.objecter ms_handle_connect 0x7f79fc06de80 2024-03-25T19:27:23.037 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:23.038+0000 7f79f7fff640 1 -- 172.21.15.83:0/1445403901 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7f7a00054720 con 0x7f7a100b1730 2024-03-25T19:27:23.038 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:23.038+0000 7f79f7fff640 10 client.4569.objecter ms_dispatch 0x7f7a10000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:27:23.038 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:23.038+0000 7f79f7fff640 3 client.4569.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:27:23.038 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:23.038+0000 7f79f7fff640 3 client.4569.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:27:23.038 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:23.038+0000 7f79f7fff640 20 client.4569.objecter dump_active .. 0 homeless 2024-03-25T19:27:23.038 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:23.038+0000 7f79f7fff640 1 -- 172.21.15.83:0/1445403901 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7a00050230 con 0x7f7a100b1730 2024-03-25T19:27:23.038 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:23.038+0000 7f79f7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:23.038 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:23.038+0000 7f79f7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:27:23.267 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:23.267+0000 7f7a15a6b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:23.267 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:23.267+0000 7f7a15a6b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:23.268 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:23.267+0000 7f7a15a6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:27:23.268 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:23.267+0000 7f7a15a6b640 1 -- 172.21.15.83:0/1445403901 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7a10143150 con 0x7f7a100b1730 2024-03-25T19:27:23.290 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:23.291+0000 7f79f7fff640 1 -- 172.21.15.83:0/1445403901 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7f7a10143150 con 0x7f7a100b1730 2024-03-25T19:27:23.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:23.291+0000 7f79f7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:23.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:23.291+0000 7f79f7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:27:23.291 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:27:23.291 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:27:23.293 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:23.293+0000 7f7a15a6b640 10 librados: watch_flush enter 2024-03-25T19:27:23.293 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:23.293+0000 7f7a15a6b640 10 librados: watch_flush exit 2024-03-25T19:27:23.294 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:23.293+0000 7f7a15a6b640 20 client.4569.objecter shutdown clearing up homeless session... 2024-03-25T19:27:23.294 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:23.293+0000 7f7a15a6b640 10 client.4569.objecter successfully canceled tick 2024-03-25T19:27:23.294 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:23.293+0000 7f7a15a6b640 1 -- 172.21.15.83:0/1445403901 >> v1:172.21.15.103:6800/345072327 conn(0x7f79fc06de80 legacy=0x7f79fc070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:23.294 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:23.293+0000 7f7a15a6b640 10 monclient: shutdown 2024-03-25T19:27:23.294 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:23.293+0000 7f7a15a6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:23.294 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:23.293+0000 7f7a15a6b640 1 -- 172.21.15.83:0/1445403901 >> v1:172.21.15.83:6789/0 conn(0x7f7a100b1730 legacy=0x7f7a1013b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:23.294 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:23.293+0000 7f7a15a6b640 1 -- 172.21.15.83:0/1445403901 shutdown_connections 2024-03-25T19:27:23.294 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:23.293+0000 7f7a15a6b640 1 -- 172.21.15.83:0/1445403901 >> 172.21.15.83:0/1445403901 conn(0x7f7a100617a0 msgr2=0x7f7a100b6460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:23.294 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:23.293+0000 7f7a15a6b640 1 -- 172.21.15.83:0/1445403901 shutdown_connections 2024-03-25T19:27:23.294 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:23.293+0000 7f7a15a6b640 1 -- 172.21.15.83:0/1445403901 wait complete. 2024-03-25T19:27:23.294 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:23.294+0000 7f7a15a6b640 1 librados: shutdown 2024-03-25T19:27:23.303 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:27:28.528 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:08:10.455 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:28.541 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:08:10.469 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:28.569 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:08:10.496 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:28.585 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:08:10.495 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:28.590 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:08:10.518 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:28.595 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:08:10.506 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:28.599 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:08:10.527 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:28.624 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:08:10.534 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:28.626 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:08:10.554 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:30.110 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:27:30.225 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.226+0000 7f5a95b85640 10 monclient: get_monmap_and_config 2024-03-25T19:27:30.225 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.226+0000 7f5a95b85640 10 monclient: build_initial_monmap 2024-03-25T19:27:30.226 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.226+0000 7f5a95b85640 10 monclient: monmap: 2024-03-25T19:27:30.226 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:27:30.226 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:27:30.226 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:27:30.227031+0000 2024-03-25T19:27:30.226 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:27:30.227031+0000 2024-03-25T19:27:30.226 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:30.226 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:27:30.226 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:27:30.226 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:27:30.226 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:27:30.226 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:27:30.228 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.227+0000 7f5a95b85640 1 Processor -- start 2024-03-25T19:27:30.228 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.227+0000 7f5a95b85640 1 -- start start 2024-03-25T19:27:30.228 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.227+0000 7f5a95b85640 10 monclient: init 2024-03-25T19:27:30.228 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.227+0000 7f5a95b85640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:30.228 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.227+0000 7f5a95b85640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:27:30.228 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.228+0000 7f5a95b85640 10 monclient(hunting): picked mon.noname-b con 0x7f5a900a98f0 addr v1:172.21.15.83:6790/0 2024-03-25T19:27:30.228 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.228+0000 7f5a95b85640 10 monclient(hunting): picked mon.noname-c con 0x7f5a9013f710 addr v1:172.21.15.103:6789/0 2024-03-25T19:27:30.228 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.228+0000 7f5a95b85640 10 monclient(hunting): picked mon.noname-a con 0x7f5a901430c0 addr v1:172.21.15.83:6789/0 2024-03-25T19:27:30.228 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.228+0000 7f5a95b85640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a90142520 con 0x7f5a901430c0 2024-03-25T19:27:30.228 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.228+0000 7f5a95b85640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a900b28a0 con 0x7f5a9013f710 2024-03-25T19:27:30.228 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.228+0000 7f5a95b85640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a900b2a30 con 0x7f5a900a98f0 2024-03-25T19:27:30.228 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.228+0000 7f5a95b85640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:30.228 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.228+0000 7f5a95b85640 10 monclient(hunting): authenticate will time out at 6507.499881s 2024-03-25T19:27:30.234 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.234+0000 7f5a8effd640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f5a9013f710 0x7f5a901429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:55010/0 (socket says 172.21.15.83:55010) 2024-03-25T19:27:30.234 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.234+0000 7f5a8effd640 1 -- 172.21.15.83:0/2164397357 learned_addr learned my addr 172.21.15.83:0/2164397357 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:27:30.242 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.242+0000 7f5a8e7fc640 1 -- 172.21.15.83:0/2164397357 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3186949187 0 0) 0x7f5a900b28a0 con 0x7f5a9013f710 2024-03-25T19:27:30.242 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.242+0000 7f5a8e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:27:30.242 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.242+0000 7f5a8e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:30.242 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.242+0000 7f5a8e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:30.242 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.242+0000 7f5a8e7fc640 10 monclient(hunting): my global_id is 4582 2024-03-25T19:27:30.242 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.242+0000 7f5a8e7fc640 1 -- 172.21.15.83:0/2164397357 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a78003520 con 0x7f5a9013f710 2024-03-25T19:27:30.250 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.249+0000 7f5a8e7fc640 1 -- 172.21.15.83:0/2164397357 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2917895035 0 0) 0x7f5a78003520 con 0x7f5a9013f710 2024-03-25T19:27:30.250 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.249+0000 7f5a8e7fc640 1 -- 172.21.15.83:0/2164397357 >> v1:172.21.15.83:6790/0 conn(0x7f5a900a98f0 legacy=0x7f5a900a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:30.250 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.250+0000 7f5a8e7fc640 1 -- 172.21.15.83:0/2164397357 >> v1:172.21.15.83:6789/0 conn(0x7f5a901430c0 legacy=0x7f5a9014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:30.250 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.250+0000 7f5a8e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:27:30.250 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.250+0000 7f5a8e7fc640 1 monclient: found mon.noname-c 2024-03-25T19:27:30.250 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.250+0000 7f5a8e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:30.250 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.250+0000 7f5a8e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.103:6789/0 2024-03-25T19:27:30.250 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.250+0000 7f5a8e7fc640 1 -- 172.21.15.83:0/2164397357 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a900b44c0 con 0x7f5a9013f710 2024-03-25T19:27:30.250 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.250+0000 7f5a8e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:27:30.251 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.250+0000 7f5a8e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:27:30.251 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.250+0000 7f5a8e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:30.251 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.250+0000 7f5a8e7fc640 1 -- 172.21.15.83:0/2164397357 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f5a84001690 con 0x7f5a9013f710 2024-03-25T19:27:30.251 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.250+0000 7f5a8e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:30.251 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.250+0000 7f5a8e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:27:30.251 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.250+0000 7f5a8e7fc640 10 monclient: dump: 2024-03-25T19:27:30.251 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:27:30.251 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:27:30.251 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:30.251 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:30.251 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:30.251 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:27:30.251 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:27:30.251 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:27:30.251 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:27:30.251 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:27:30.252 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.250+0000 7f5a95b85640 5 monclient: authenticate success, global_id 4582 2024-03-25T19:27:30.252 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.250+0000 7f5a95b85640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:27:30.260 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.260+0000 7f5a8e7fc640 1 -- 172.21.15.83:0/2164397357 <== mon.1 v1:172.21.15.103:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a840017f0 con 0x7f5a9013f710 2024-03-25T19:27:30.260 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.260+0000 7f5a8e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:30.260 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.260+0000 7f5a8e7fc640 1 -- 172.21.15.83:0/2164397357 <== mon.1 v1:172.21.15.103:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f5a84001c00 con 0x7f5a9013f710 2024-03-25T19:27:30.260 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.260+0000 7f5a95b85640 10 monclient: get_monmap_and_config success 2024-03-25T19:27:30.260 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.260+0000 7f5a95b85640 4 set_mon_vals no callback set 2024-03-25T19:27:30.260 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.260+0000 7f5a8e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:30.260 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.260+0000 7f5a8e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:27:30.260 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.260+0000 7f5a8e7fc640 10 monclient: dump: 2024-03-25T19:27:30.260 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:27:30.260 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:27:30.261 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:30.261 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:30.261 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:30.261 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:27:30.261 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:27:30.261 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:27:30.261 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:27:30.261 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:27:30.262 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.261+0000 7f5a95b85640 10 monclient: shutdown 2024-03-25T19:27:30.262 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.261+0000 7f5a95b85640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:30.262 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.261+0000 7f5a95b85640 1 -- 172.21.15.83:0/2164397357 >> v1:172.21.15.103:6789/0 conn(0x7f5a9013f710 legacy=0x7f5a901429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:30.262 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.261+0000 7f5a95b85640 1 -- 172.21.15.83:0/2164397357 shutdown_connections 2024-03-25T19:27:30.262 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.261+0000 7f5a95b85640 1 -- 172.21.15.83:0/2164397357 >> 172.21.15.83:0/2164397357 conn(0x7f5a900b7ea0 msgr2=0x7f5a900b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:30.262 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.262+0000 7f5a95b85640 1 -- 172.21.15.83:0/2164397357 shutdown_connections 2024-03-25T19:27:30.262 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.262+0000 7f5a95b85640 1 -- 172.21.15.83:0/2164397357 wait complete. 2024-03-25T19:27:30.263 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.263+0000 7f5a95b85640 10 monclient: build_initial_monmap 2024-03-25T19:27:30.263 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.263+0000 7f5a95b85640 10 monclient: monmap: 2024-03-25T19:27:30.263 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:27:30.263 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:27:30.263 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:27:30.263 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:27:30.263 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:30.264 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:27:30.264 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:27:30.264 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:27:30.264 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:27:30.264 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:27:30.264 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.263+0000 7f5a95b85640 1 librados: starting msgr at 2024-03-25T19:27:30.264 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.263+0000 7f5a95b85640 1 librados: starting objecter 2024-03-25T19:27:30.264 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.263+0000 7f5a95b85640 1 Processor -- start 2024-03-25T19:27:30.264 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.264+0000 7f5a95b85640 1 -- start start 2024-03-25T19:27:30.264 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.264+0000 7f5a95b85640 1 librados: setting wanted keys 2024-03-25T19:27:30.264 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.264+0000 7f5a95b85640 1 librados: calling monclient init 2024-03-25T19:27:30.264 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.264+0000 7f5a95b85640 10 monclient: init 2024-03-25T19:27:30.265 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.264+0000 7f5a95b85640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:30.265 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.264+0000 7f5a95b85640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:27:30.265 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.264+0000 7f5a95b85640 10 monclient(hunting): picked mon.noname-a con 0x7f5a900a98f0 addr v1:172.21.15.83:6789/0 2024-03-25T19:27:30.265 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.264+0000 7f5a95b85640 10 monclient(hunting): picked mon.noname-c con 0x7f5a9013f710 addr v1:172.21.15.83:6790/0 2024-03-25T19:27:30.265 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.264+0000 7f5a95b85640 10 monclient(hunting): picked mon.noname-b con 0x7f5a901430c0 addr v1:172.21.15.103:6789/0 2024-03-25T19:27:30.265 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.264+0000 7f5a95b85640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a9013be70 con 0x7f5a900a98f0 2024-03-25T19:27:30.265 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.264+0000 7f5a95b85640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a90131660 con 0x7f5a901430c0 2024-03-25T19:27:30.265 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.264+0000 7f5a95b85640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a901317f0 con 0x7f5a9013f710 2024-03-25T19:27:30.266 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.264+0000 7f5a95b85640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:30.266 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.264+0000 7f5a95b85640 10 monclient(hunting): authenticate will time out at 6507.536678s 2024-03-25T19:27:30.269 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.269+0000 7f5a8effd640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f5a9013f710 0x7f5a901d76e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:45928/0 (socket says 172.21.15.83:45928) 2024-03-25T19:27:30.269 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.269+0000 7f5a8effd640 1 -- 172.21.15.83:0/2061548923 learned_addr learned my addr 172.21.15.83:0/2061548923 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:27:30.272 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.272+0000 7f5a77fff640 10 client.?.objecter ms_handle_connect 0x7f5a9013f710 2024-03-25T19:27:30.272 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.272+0000 7f5a77fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:30.278 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.278+0000 7f5a77fff640 10 client.?.objecter ms_handle_connect 0x7f5a901430c0 2024-03-25T19:27:30.278 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.278+0000 7f5a77fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:30.282 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.282+0000 7f5a77fff640 1 -- 172.21.15.83:0/2061548923 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3305040031 0 0) 0x7f5a901317f0 con 0x7f5a9013f710 2024-03-25T19:27:30.282 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.282+0000 7f5a77fff640 10 monclient(hunting): _negotiate 2024-03-25T19:27:30.282 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.282+0000 7f5a77fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:30.282 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.282+0000 7f5a77fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:30.282 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.282+0000 7f5a77fff640 10 monclient(hunting): my global_id is 4589 2024-03-25T19:27:30.282 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.282+0000 7f5a77fff640 1 -- 172.21.15.83:0/2061548923 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a7c004eb0 con 0x7f5a9013f710 2024-03-25T19:27:30.284 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.285+0000 7f5a77fff640 10 client.?.objecter ms_handle_connect 0x7f5a900a98f0 2024-03-25T19:27:30.284 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.285+0000 7f5a77fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:30.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.291+0000 7f5a77fff640 1 -- 172.21.15.83:0/2061548923 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3460802060 0 0) 0x7f5a90131660 con 0x7f5a901430c0 2024-03-25T19:27:30.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.291+0000 7f5a77fff640 10 monclient(hunting): _negotiate 2024-03-25T19:27:30.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.291+0000 7f5a77fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:30.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.291+0000 7f5a77fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:30.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.291+0000 7f5a77fff640 10 monclient(hunting): my global_id is 4585 2024-03-25T19:27:30.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.291+0000 7f5a77fff640 1 -- 172.21.15.83:0/2061548923 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a901317f0 con 0x7f5a901430c0 2024-03-25T19:27:30.294 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.294+0000 7f5a77fff640 1 -- 172.21.15.83:0/2061548923 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2722687228 0 0) 0x7f5a9013be70 con 0x7f5a900a98f0 2024-03-25T19:27:30.294 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.295+0000 7f5a77fff640 10 monclient(hunting): _negotiate 2024-03-25T19:27:30.295 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.295+0000 7f5a77fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:30.295 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.295+0000 7f5a77fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:30.295 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.295+0000 7f5a77fff640 10 monclient(hunting): my global_id is 4575 2024-03-25T19:27:30.295 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.295+0000 7f5a77fff640 1 -- 172.21.15.83:0/2061548923 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a90131660 con 0x7f5a900a98f0 2024-03-25T19:27:30.297 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.297+0000 7f5a77fff640 1 -- 172.21.15.83:0/2061548923 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 440284482 0 0) 0x7f5a7c004eb0 con 0x7f5a9013f710 2024-03-25T19:27:30.297 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.297+0000 7f5a77fff640 1 -- 172.21.15.83:0/2061548923 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a9013be70 con 0x7f5a9013f710 2024-03-25T19:27:30.297 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.297+0000 7f5a77fff640 1 -- 172.21.15.83:0/2061548923 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f5a84001c00 con 0x7f5a9013f710 2024-03-25T19:27:30.297 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.297+0000 7f5a77fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:30.297 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.297+0000 7f5a77fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:27:30.297 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.297+0000 7f5a77fff640 10 monclient(hunting): dump: 2024-03-25T19:27:30.297 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:27:30.297 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:27:30.298 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:30.298 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:30.298 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:30.298 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:27:30.298 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:27:30.298 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:27:30.298 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:27:30.298 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:27:30.298 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.297+0000 7f5a77fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:27:30.301 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.301+0000 7f5a77fff640 1 -- 172.21.15.83:0/2061548923 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3160948573 0 0) 0x7f5a901317f0 con 0x7f5a901430c0 2024-03-25T19:27:30.301 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.301+0000 7f5a77fff640 1 -- 172.21.15.83:0/2061548923 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a7c004eb0 con 0x7f5a901430c0 2024-03-25T19:27:30.301 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.301+0000 7f5a77fff640 1 -- 172.21.15.83:0/2061548923 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f5a88001b00 con 0x7f5a901430c0 2024-03-25T19:27:30.301 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.301+0000 7f5a77fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:30.301 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.301+0000 7f5a77fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:27:30.301 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.301+0000 7f5a77fff640 10 monclient(hunting): dump: 2024-03-25T19:27:30.301 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:27:30.301 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:27:30.301 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:30.301 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:30.301 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:30.301 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:27:30.302 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:27:30.302 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:27:30.302 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:27:30.302 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:27:30.309 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.309+0000 7f5a77fff640 1 -- 172.21.15.83:0/2061548923 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4073724224 0 0) 0x7f5a90131660 con 0x7f5a900a98f0 2024-03-25T19:27:30.309 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.309+0000 7f5a77fff640 1 -- 172.21.15.83:0/2061548923 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a901317f0 con 0x7f5a900a98f0 2024-03-25T19:27:30.309 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.309+0000 7f5a77fff640 1 -- 172.21.15.83:0/2061548923 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f5a80001830 con 0x7f5a900a98f0 2024-03-25T19:27:30.309 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.309+0000 7f5a77fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:30.309 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.309+0000 7f5a77fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:27:30.309 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.309+0000 7f5a77fff640 10 monclient(hunting): dump: 2024-03-25T19:27:30.309 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:27:30.309 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:27:30.309 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:30.310 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:30.310 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:30.310 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:27:30.310 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:27:30.310 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:27:30.310 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:27:30.310 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:27:30.311 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.310+0000 7f5a77fff640 1 -- 172.21.15.83:0/2061548923 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3460349764 0 0) 0x7f5a9013be70 con 0x7f5a9013f710 2024-03-25T19:27:30.311 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.310+0000 7f5a77fff640 1 -- 172.21.15.83:0/2061548923 >> v1:172.21.15.103:6789/0 conn(0x7f5a901430c0 legacy=0x7f5a901e2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:30.311 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.310+0000 7f5a77fff640 1 -- 172.21.15.83:0/2061548923 >> v1:172.21.15.83:6789/0 conn(0x7f5a900a98f0 legacy=0x7f5a90130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:30.311 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.310+0000 7f5a77fff640 10 monclient: _finish_hunting 0 2024-03-25T19:27:30.312 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.310+0000 7f5a77fff640 1 monclient: found mon.c 2024-03-25T19:27:30.312 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.310+0000 7f5a77fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:30.312 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.310+0000 7f5a77fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:27:30.312 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.310+0000 7f5a77fff640 1 -- 172.21.15.83:0/2061548923 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a90131980 con 0x7f5a9013f710 2024-03-25T19:27:30.312 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.310+0000 7f5a77fff640 10 monclient: _finish_auth 0 2024-03-25T19:27:30.312 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.310+0000 7f5a77fff640 10 monclient: _check_auth_tickets 2024-03-25T19:27:30.312 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.310+0000 7f5a77fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:30.312 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.310+0000 7f5a95b85640 5 monclient: authenticate success, global_id 4589 2024-03-25T19:27:30.312 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.310+0000 7f5a95b85640 10 monclient: _renew_subs 2024-03-25T19:27:30.312 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.310+0000 7f5a95b85640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:27:30.312 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.310+0000 7f5a95b85640 1 -- 172.21.15.83:0/2061548923 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a90135c60 con 0x7f5a9013f710 2024-03-25T19:27:30.312 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.310+0000 7f5a95b85640 10 client.4589.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:27:30.312 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.310+0000 7f5a95b85640 10 monclient: _renew_subs 2024-03-25T19:27:30.312 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.310+0000 7f5a95b85640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:27:30.313 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.310+0000 7f5a95b85640 1 -- 172.21.15.83:0/2061548923 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a900b7770 con 0x7f5a9013f710 2024-03-25T19:27:30.313 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.310+0000 7f5a95b85640 1 librados: init done 2024-03-25T19:27:30.313 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.311+0000 7f5a95b85640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:30.313 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.311+0000 7f5a95b85640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:30.313 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.311+0000 7f5a95b85640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:27:30.313 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.311+0000 7f5a95b85640 1 -- 172.21.15.83:0/2061548923 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a90074cc0 con 0x7f5a9013f710 2024-03-25T19:27:30.611 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.611+0000 7f5a77fff640 1 -- 172.21.15.83:0/2061548923 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a84004a90 con 0x7f5a9013f710 2024-03-25T19:27:30.612 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.611+0000 7f5a77fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:30.612 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.611+0000 7f5a77fff640 1 -- 172.21.15.83:0/2061548923 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f5a84004e20 con 0x7f5a9013f710 2024-03-25T19:27:30.612 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.612+0000 7f5a77fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:30.612 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.612+0000 7f5a77fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:27:30.612 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.612+0000 7f5a77fff640 10 monclient: dump: 2024-03-25T19:27:30.612 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:27:30.612 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:27:30.612 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:30.612 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:30.612 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:30.612 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:27:30.612 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:27:30.613 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:27:30.613 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:27:30.613 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:27:30.613 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.612+0000 7f5a8dffb640 4 set_mon_vals no callback set 2024-03-25T19:27:30.613 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.612+0000 7f5a77fff640 1 -- 172.21.15.83:0/2061548923 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7f5a84005040 con 0x7f5a9013f710 2024-03-25T19:27:30.613 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.613+0000 7f5a8f7fe640 10 monclient: get_auth_request con 0x7f5a7c06e180 auth_method 0 2024-03-25T19:27:30.614 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.613+0000 7f5a77fff640 10 client.4589.objecter ms_handle_connect 0x7f5a7c06e180 2024-03-25T19:27:30.618 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.618+0000 7f5a77fff640 1 -- 172.21.15.83:0/2061548923 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7f5a84054970 con 0x7f5a9013f710 2024-03-25T19:27:30.618 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.618+0000 7f5a77fff640 10 client.4589.objecter ms_dispatch 0x7f5a90000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:27:30.618 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.618+0000 7f5a77fff640 3 client.4589.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:27:30.618 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.618+0000 7f5a77fff640 3 client.4589.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:27:30.618 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.618+0000 7f5a77fff640 20 client.4589.objecter dump_active .. 0 homeless 2024-03-25T19:27:30.618 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.618+0000 7f5a77fff640 1 -- 172.21.15.83:0/2061548923 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5a84050480 con 0x7f5a9013f710 2024-03-25T19:27:30.618 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.618+0000 7f5a77fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:30.618 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.618+0000 7f5a77fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:27:30.868 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.868+0000 7f5a95b85640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:30.868 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.868+0000 7f5a95b85640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:30.868 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.868+0000 7f5a95b85640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:27:30.868 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.868+0000 7f5a95b85640 1 -- 172.21.15.83:0/2061548923 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a901371d0 con 0x7f5a9013f710 2024-03-25T19:27:30.918 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.918+0000 7f5a77fff640 1 -- 172.21.15.83:0/2061548923 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7f5a840523b0 con 0x7f5a9013f710 2024-03-25T19:27:30.918 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.918+0000 7f5a77fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:30.918 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.918+0000 7f5a77fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:27:30.918 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:27:30.918 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:27:30.921 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.920+0000 7f5a95b85640 10 librados: watch_flush enter 2024-03-25T19:27:30.921 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.920+0000 7f5a95b85640 10 librados: watch_flush exit 2024-03-25T19:27:30.921 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.920+0000 7f5a95b85640 20 client.4589.objecter shutdown clearing up homeless session... 2024-03-25T19:27:30.921 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.920+0000 7f5a95b85640 10 client.4589.objecter successfully canceled tick 2024-03-25T19:27:30.921 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.920+0000 7f5a95b85640 1 -- 172.21.15.83:0/2061548923 >> v1:172.21.15.103:6800/345072327 conn(0x7f5a7c06e180 legacy=0x7f5a7c070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:30.921 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.920+0000 7f5a95b85640 10 monclient: shutdown 2024-03-25T19:27:30.921 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.920+0000 7f5a95b85640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:30.921 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.920+0000 7f5a95b85640 1 -- 172.21.15.83:0/2061548923 >> v1:172.21.15.83:6790/0 conn(0x7f5a9013f710 legacy=0x7f5a901d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:30.921 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.920+0000 7f5a95b85640 1 -- 172.21.15.83:0/2061548923 shutdown_connections 2024-03-25T19:27:30.921 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.920+0000 7f5a95b85640 1 -- 172.21.15.83:0/2061548923 >> 172.21.15.83:0/2061548923 conn(0x7f5a900b7ea0 msgr2=0x7f5a90074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:30.921 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.921+0000 7f5a95b85640 1 -- 172.21.15.83:0/2061548923 shutdown_connections 2024-03-25T19:27:30.921 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.921+0000 7f5a95b85640 1 -- 172.21.15.83:0/2061548923 wait complete. 2024-03-25T19:27:30.921 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:30.921+0000 7f5a95b85640 1 librados: shutdown 2024-03-25T19:27:30.931 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:27:37.738 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:27:37.854 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.854+0000 7fcbc8338640 10 monclient: get_monmap_and_config 2024-03-25T19:27:37.855 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.854+0000 7fcbc8338640 10 monclient: build_initial_monmap 2024-03-25T19:27:37.855 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.854+0000 7fcbc8338640 10 monclient: monmap: 2024-03-25T19:27:37.855 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:27:37.855 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:27:37.855 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:27:37.855950+0000 2024-03-25T19:27:37.855 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:27:37.855950+0000 2024-03-25T19:27:37.855 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:37.855 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:27:37.855 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:27:37.855 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:27:37.855 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:27:37.855 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:27:37.857 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.854+0000 7fcbc8338640 1 Processor -- start 2024-03-25T19:27:37.857 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.854+0000 7fcbc8338640 1 -- start start 2024-03-25T19:27:37.857 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.854+0000 7fcbc8338640 10 monclient: init 2024-03-25T19:27:37.857 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.855+0000 7fcbc8338640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:37.857 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.855+0000 7fcbc8338640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:27:37.857 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.855+0000 7fcbc8338640 10 monclient(hunting): picked mon.noname-a con 0x7fcbc00a98f0 addr v1:172.21.15.83:6789/0 2024-03-25T19:27:37.857 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.855+0000 7fcbc8338640 10 monclient(hunting): picked mon.noname-b con 0x7fcbc013f710 addr v1:172.21.15.83:6790/0 2024-03-25T19:27:37.857 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.855+0000 7fcbc8338640 10 monclient(hunting): picked mon.noname-c con 0x7fcbc01430c0 addr v1:172.21.15.103:6789/0 2024-03-25T19:27:37.857 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.855+0000 7fcbc8338640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbc0142520 con 0x7fcbc00a98f0 2024-03-25T19:27:37.857 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.855+0000 7fcbc8338640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbc00b28a0 con 0x7fcbc01430c0 2024-03-25T19:27:37.857 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.855+0000 7fcbc8338640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbc00b2a30 con 0x7fcbc013f710 2024-03-25T19:27:37.857 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.855+0000 7fcbc8338640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:37.857 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.855+0000 7fcbc8338640 10 monclient(hunting): authenticate will time out at 6515.128431s 2024-03-25T19:27:37.860 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.860+0000 7fcbc60d4640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fcbc00a98f0 0x7fcbc00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:33128/0 (socket says 172.21.15.83:33128) 2024-03-25T19:27:37.860 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.860+0000 7fcbc60d4640 1 -- 172.21.15.83:0/3651884901 learned_addr learned my addr 172.21.15.83:0/3651884901 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:27:37.879 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.879+0000 7fcbc50d2640 1 -- 172.21.15.83:0/3651884901 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3114385788 0 0) 0x7fcbc0142520 con 0x7fcbc00a98f0 2024-03-25T19:27:37.879 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.879+0000 7fcbc50d2640 10 monclient(hunting): _negotiate 2024-03-25T19:27:37.879 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.879+0000 7fcbc50d2640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:37.879 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.879+0000 7fcbc50d2640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:37.879 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.879+0000 7fcbc50d2640 10 monclient(hunting): my global_id is 4578 2024-03-25T19:27:37.879 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.879+0000 7fcbc50d2640 1 -- 172.21.15.83:0/3651884901 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbac003520 con 0x7fcbc00a98f0 2024-03-25T19:27:37.887 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.887+0000 7fcbc50d2640 1 -- 172.21.15.83:0/3651884901 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2892880428 0 0) 0x7fcbc00b2a30 con 0x7fcbc013f710 2024-03-25T19:27:37.888 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.887+0000 7fcbc50d2640 10 monclient(hunting): _negotiate 2024-03-25T19:27:37.888 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.887+0000 7fcbc50d2640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:37.888 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.887+0000 7fcbc50d2640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:37.888 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.887+0000 7fcbc50d2640 10 monclient(hunting): my global_id is 4592 2024-03-25T19:27:37.888 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.887+0000 7fcbc50d2640 1 -- 172.21.15.83:0/3651884901 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbc0142520 con 0x7fcbc013f710 2024-03-25T19:27:37.898 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.897+0000 7fcbc50d2640 1 -- 172.21.15.83:0/3651884901 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2684659650 0 0) 0x7fcbc00b28a0 con 0x7fcbc01430c0 2024-03-25T19:27:37.898 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.897+0000 7fcbc50d2640 10 monclient(hunting): _negotiate 2024-03-25T19:27:37.898 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.897+0000 7fcbc50d2640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:37.898 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.897+0000 7fcbc50d2640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:37.898 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.897+0000 7fcbc50d2640 10 monclient(hunting): my global_id is 4588 2024-03-25T19:27:37.898 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.897+0000 7fcbc50d2640 1 -- 172.21.15.83:0/3651884901 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbc00b2a30 con 0x7fcbc01430c0 2024-03-25T19:27:37.898 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.897+0000 7fcbc50d2640 1 -- 172.21.15.83:0/3651884901 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2486232695 0 0) 0x7fcbac003520 con 0x7fcbc00a98f0 2024-03-25T19:27:37.899 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.897+0000 7fcbc50d2640 1 -- 172.21.15.83:0/3651884901 >> v1:172.21.15.83:6790/0 conn(0x7fcbc013f710 legacy=0x7fcbc01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:37.899 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.897+0000 7fcbc50d2640 1 -- 172.21.15.83:0/3651884901 >> v1:172.21.15.103:6789/0 conn(0x7fcbc01430c0 legacy=0x7fcbc014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:37.899 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.897+0000 7fcbc50d2640 10 monclient: _finish_hunting 0 2024-03-25T19:27:37.899 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.897+0000 7fcbc50d2640 1 monclient: found mon.noname-a 2024-03-25T19:27:37.899 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.897+0000 7fcbc50d2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:37.899 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.897+0000 7fcbc50d2640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.83:6789/0 2024-03-25T19:27:37.899 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.897+0000 7fcbc50d2640 1 -- 172.21.15.83:0/3651884901 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbc00b44c0 con 0x7fcbc00a98f0 2024-03-25T19:27:37.899 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.897+0000 7fcbc50d2640 10 monclient: _finish_auth 0 2024-03-25T19:27:37.899 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.897+0000 7fcbc50d2640 10 monclient: _check_auth_tickets 2024-03-25T19:27:37.899 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.897+0000 7fcbc50d2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:37.899 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.897+0000 7fcbc50d2640 1 -- 172.21.15.83:0/3651884901 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fcbbc001690 con 0x7fcbc00a98f0 2024-03-25T19:27:37.899 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.897+0000 7fcbc8338640 5 monclient: authenticate success, global_id 4578 2024-03-25T19:27:37.899 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.897+0000 7fcbc50d2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:37.899 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.897+0000 7fcbc50d2640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:27:37.899 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.897+0000 7fcbc50d2640 10 monclient: dump: 2024-03-25T19:27:37.900 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:27:37.900 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:27:37.900 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:37.900 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:37.900 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:37.900 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:27:37.900 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:27:37.900 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:27:37.900 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:27:37.900 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:27:37.900 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.897+0000 7fcbc8338640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:27:37.922 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.921+0000 7fcbc50d2640 1 -- 172.21.15.83:0/3651884901 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcbbc001870 con 0x7fcbc00a98f0 2024-03-25T19:27:37.922 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.921+0000 7fcbc50d2640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:37.922 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.921+0000 7fcbc50d2640 1 -- 172.21.15.83:0/3651884901 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fcbbc001c80 con 0x7fcbc00a98f0 2024-03-25T19:27:37.922 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.921+0000 7fcbc8338640 10 monclient: get_monmap_and_config success 2024-03-25T19:27:37.923 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.922+0000 7fcbc8338640 4 set_mon_vals no callback set 2024-03-25T19:27:37.923 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.922+0000 7fcbc50d2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:37.923 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.922+0000 7fcbc50d2640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:27:37.923 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.922+0000 7fcbc50d2640 10 monclient: dump: 2024-03-25T19:27:37.923 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:27:37.923 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:27:37.923 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:37.923 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:37.923 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:37.923 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:27:37.923 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:27:37.923 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:27:37.923 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:27:37.923 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:27:37.924 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.922+0000 7fcbc8338640 10 monclient: shutdown 2024-03-25T19:27:37.925 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.922+0000 7fcbc8338640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:37.925 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.922+0000 7fcbc8338640 1 -- 172.21.15.83:0/3651884901 >> v1:172.21.15.83:6789/0 conn(0x7fcbc00a98f0 legacy=0x7fcbc00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:37.925 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.922+0000 7fcbc8338640 1 -- 172.21.15.83:0/3651884901 shutdown_connections 2024-03-25T19:27:37.925 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.922+0000 7fcbc8338640 1 -- 172.21.15.83:0/3651884901 >> 172.21.15.83:0/3651884901 conn(0x7fcbc00b7ea0 msgr2=0x7fcbc00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:37.925 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.922+0000 7fcbc8338640 1 -- 172.21.15.83:0/3651884901 shutdown_connections 2024-03-25T19:27:37.925 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.922+0000 7fcbc8338640 1 -- 172.21.15.83:0/3651884901 wait complete. 2024-03-25T19:27:37.925 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.923+0000 7fcbc8338640 10 monclient: build_initial_monmap 2024-03-25T19:27:37.925 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.923+0000 7fcbc8338640 10 monclient: monmap: 2024-03-25T19:27:37.925 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:27:37.925 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:27:37.925 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:27:37.925 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:27:37.925 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:37.925 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:27:37.926 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:27:37.926 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:27:37.926 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:27:37.926 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:27:37.926 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.924+0000 7fcbc8338640 1 librados: starting msgr at 2024-03-25T19:27:37.926 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.924+0000 7fcbc8338640 1 librados: starting objecter 2024-03-25T19:27:37.926 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.924+0000 7fcbc8338640 1 Processor -- start 2024-03-25T19:27:37.926 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.924+0000 7fcbc8338640 1 -- start start 2024-03-25T19:27:37.926 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.924+0000 7fcbc8338640 1 librados: setting wanted keys 2024-03-25T19:27:37.926 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.924+0000 7fcbc8338640 1 librados: calling monclient init 2024-03-25T19:27:37.926 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.924+0000 7fcbc8338640 10 monclient: init 2024-03-25T19:27:37.927 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.924+0000 7fcbc8338640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:37.927 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.924+0000 7fcbc8338640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:27:37.927 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.924+0000 7fcbc8338640 10 monclient(hunting): picked mon.noname-c con 0x7fcbc00a98f0 addr v1:172.21.15.83:6790/0 2024-03-25T19:27:37.928 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.924+0000 7fcbc8338640 10 monclient(hunting): picked mon.noname-a con 0x7fcbc013f710 addr v1:172.21.15.83:6789/0 2024-03-25T19:27:37.928 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.924+0000 7fcbc8338640 10 monclient(hunting): picked mon.noname-b con 0x7fcbc01430c0 addr v1:172.21.15.103:6789/0 2024-03-25T19:27:37.928 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.924+0000 7fcbc8338640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbc0132a30 con 0x7fcbc013f710 2024-03-25T19:27:37.928 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.925+0000 7fcbc8338640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbc0138c80 con 0x7fcbc01430c0 2024-03-25T19:27:37.928 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.925+0000 7fcbc8338640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbc0138f60 con 0x7fcbc00a98f0 2024-03-25T19:27:37.928 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.925+0000 7fcbc8338640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:37.928 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.925+0000 7fcbc8338640 10 monclient(hunting): authenticate will time out at 6515.197831s 2024-03-25T19:27:37.936 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.936+0000 7fcbc68d5640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fcbc01430c0 0x7fcbc0132320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:55924/0 (socket says 172.21.15.83:55924) 2024-03-25T19:27:37.936 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.936+0000 7fcbc68d5640 1 -- 172.21.15.83:0/4235838608 learned_addr learned my addr 172.21.15.83:0/4235838608 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:27:37.937 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.936+0000 7fcbb6ffd640 10 client.?.objecter ms_handle_connect 0x7fcbc01430c0 2024-03-25T19:27:37.937 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.936+0000 7fcbb6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:37.941 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.941+0000 7fcbb6ffd640 10 client.?.objecter ms_handle_connect 0x7fcbc013f710 2024-03-25T19:27:37.941 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.941+0000 7fcbb6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:37.949 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.948+0000 7fcbb6ffd640 10 client.?.objecter ms_handle_connect 0x7fcbc00a98f0 2024-03-25T19:27:37.949 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.948+0000 7fcbb6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:37.958 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.958+0000 7fcbb6ffd640 1 -- 172.21.15.83:0/4235838608 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3417611858 0 0) 0x7fcbc0132a30 con 0x7fcbc013f710 2024-03-25T19:27:37.958 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.958+0000 7fcbb6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:27:37.959 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.958+0000 7fcbb6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:37.959 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.958+0000 7fcbb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:37.959 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.958+0000 7fcbb6ffd640 10 monclient(hunting): my global_id is 4581 2024-03-25T19:27:37.959 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.958+0000 7fcbb6ffd640 1 -- 172.21.15.83:0/4235838608 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcba4004eb0 con 0x7fcbc013f710 2024-03-25T19:27:37.963 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.962+0000 7fcbb6ffd640 1 -- 172.21.15.83:0/4235838608 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2841185211 0 0) 0x7fcbc0138c80 con 0x7fcbc01430c0 2024-03-25T19:27:37.963 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.962+0000 7fcbb6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:27:37.963 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.962+0000 7fcbb6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:37.963 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.962+0000 7fcbb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:37.963 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.962+0000 7fcbb6ffd640 10 monclient(hunting): my global_id is 4591 2024-03-25T19:27:37.963 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.962+0000 7fcbb6ffd640 1 -- 172.21.15.83:0/4235838608 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbc0132a30 con 0x7fcbc01430c0 2024-03-25T19:27:37.963 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.962+0000 7fcbb6ffd640 1 -- 172.21.15.83:0/4235838608 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3450268570 0 0) 0x7fcbc0138f60 con 0x7fcbc00a98f0 2024-03-25T19:27:37.964 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.962+0000 7fcbb6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:27:37.964 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.962+0000 7fcbb6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:37.964 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.962+0000 7fcbb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:37.964 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.962+0000 7fcbb6ffd640 10 monclient(hunting): my global_id is 4595 2024-03-25T19:27:37.964 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.962+0000 7fcbb6ffd640 1 -- 172.21.15.83:0/4235838608 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbc0138c80 con 0x7fcbc00a98f0 2024-03-25T19:27:37.981 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.980+0000 7fcbb6ffd640 1 -- 172.21.15.83:0/4235838608 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1379082314 0 0) 0x7fcbc0132a30 con 0x7fcbc01430c0 2024-03-25T19:27:37.981 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.981+0000 7fcbb6ffd640 1 -- 172.21.15.83:0/4235838608 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcbc0138f60 con 0x7fcbc01430c0 2024-03-25T19:27:37.981 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.981+0000 7fcbb6ffd640 1 -- 172.21.15.83:0/4235838608 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fcbb8001c50 con 0x7fcbc01430c0 2024-03-25T19:27:37.982 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.981+0000 7fcbb6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:37.982 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.981+0000 7fcbb6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:27:37.982 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.981+0000 7fcbb6ffd640 10 monclient(hunting): dump: 2024-03-25T19:27:37.982 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:27:37.982 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:27:37.982 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:37.982 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:37.982 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:37.982 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:27:37.982 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:27:37.982 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:27:37.982 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:27:37.982 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:27:37.982 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.981+0000 7fcbb6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:27:37.991 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.990+0000 7fcbb6ffd640 1 -- 172.21.15.83:0/4235838608 <== mon.1 v1:172.21.15.103:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4006925591 0 0) 0x7fcbc0138f60 con 0x7fcbc01430c0 2024-03-25T19:27:37.991 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.990+0000 7fcbb6ffd640 1 -- 172.21.15.83:0/4235838608 >> v1:172.21.15.83:6790/0 conn(0x7fcbc00a98f0 legacy=0x7fcbc0137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:37.991 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.990+0000 7fcbb6ffd640 1 -- 172.21.15.83:0/4235838608 >> v1:172.21.15.83:6789/0 conn(0x7fcbc013f710 legacy=0x7fcbc0131c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:37.991 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.990+0000 7fcbb6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:27:37.991 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.990+0000 7fcbb6ffd640 1 monclient: found mon.b 2024-03-25T19:27:37.992 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.990+0000 7fcbb6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:37.992 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.990+0000 7fcbb6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:27:37.992 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.990+0000 7fcbb6ffd640 1 -- 172.21.15.83:0/4235838608 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbc0139240 con 0x7fcbc01430c0 2024-03-25T19:27:37.992 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.990+0000 7fcbb6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:27:37.992 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.990+0000 7fcbb6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:27:37.992 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.990+0000 7fcbb6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:37.992 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.990+0000 7fcbc8338640 5 monclient: authenticate success, global_id 4591 2024-03-25T19:27:37.992 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.990+0000 7fcbc8338640 10 monclient: _renew_subs 2024-03-25T19:27:37.992 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.990+0000 7fcbc8338640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:27:37.992 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.990+0000 7fcbc8338640 1 -- 172.21.15.83:0/4235838608 --> v1:172.21.15.103:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbc0136e50 con 0x7fcbc01430c0 2024-03-25T19:27:37.992 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.990+0000 7fcbc8338640 10 client.4591.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:27:37.992 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.990+0000 7fcbc8338640 10 monclient: _renew_subs 2024-03-25T19:27:37.992 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.990+0000 7fcbc8338640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:27:37.992 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.990+0000 7fcbc8338640 1 -- 172.21.15.83:0/4235838608 --> v1:172.21.15.103:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbc01389f0 con 0x7fcbc01430c0 2024-03-25T19:27:37.992 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.990+0000 7fcbc8338640 1 librados: init done 2024-03-25T19:27:37.993 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.991+0000 7fcbc8338640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:37.993 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.991+0000 7fcbc8338640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:37.993 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.991+0000 7fcbc8338640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:27:37.993 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:37.991+0000 7fcbc8338640 1 -- 172.21.15.83:0/4235838608 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbc00627b0 con 0x7fcbc01430c0 2024-03-25T19:27:38.276 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:38.275+0000 7fcbb6ffd640 1 -- 172.21.15.83:0/4235838608 <== mon.1 v1:172.21.15.103:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcbb80020e0 con 0x7fcbc01430c0 2024-03-25T19:27:38.277 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:38.275+0000 7fcbb6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:38.277 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:38.276+0000 7fcbb6ffd640 1 -- 172.21.15.83:0/4235838608 <== mon.1 v1:172.21.15.103:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fcbb8003b10 con 0x7fcbc01430c0 2024-03-25T19:27:38.277 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:38.276+0000 7fcbb6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:38.277 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:38.276+0000 7fcbb6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:27:38.277 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:38.276+0000 7fcbb6ffd640 10 monclient: dump: 2024-03-25T19:27:38.277 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:27:38.277 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:27:38.277 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:38.277 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:38.277 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:38.277 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:27:38.277 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:27:38.278 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:27:38.278 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:27:38.278 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:27:38.278 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:38.276+0000 7fcbc48d1640 4 set_mon_vals no callback set 2024-03-25T19:27:38.278 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:38.276+0000 7fcbb6ffd640 1 -- 172.21.15.83:0/4235838608 <== mon.1 v1:172.21.15.103:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7fcbb8003d30 con 0x7fcbc01430c0 2024-03-25T19:27:38.278 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:38.277+0000 7fcbc60d4640 10 monclient: get_auth_request con 0x7fcba406de20 auth_method 0 2024-03-25T19:27:38.279 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:38.277+0000 7fcbb6ffd640 10 client.4591.objecter ms_handle_connect 0x7fcba406de20 2024-03-25T19:27:38.282 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:38.282+0000 7fcbb6ffd640 1 -- 172.21.15.83:0/4235838608 <== mon.1 v1:172.21.15.103:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7fcbb8054a80 con 0x7fcbc01430c0 2024-03-25T19:27:38.283 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:38.282+0000 7fcbb6ffd640 10 client.4591.objecter ms_dispatch 0x7fcbc0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:27:38.283 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:38.282+0000 7fcbb6ffd640 3 client.4591.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:27:38.283 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:38.282+0000 7fcbb6ffd640 3 client.4591.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:27:38.283 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:38.282+0000 7fcbb6ffd640 20 client.4591.objecter dump_active .. 0 homeless 2024-03-25T19:27:38.283 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:38.282+0000 7fcbb6ffd640 1 -- 172.21.15.83:0/4235838608 <== mon.1 v1:172.21.15.103:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcbb80503c0 con 0x7fcbc01430c0 2024-03-25T19:27:38.283 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:38.283+0000 7fcbb6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:38.283 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:38.283+0000 7fcbb6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:27:38.496 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:38.495+0000 7fcbc8338640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:38.496 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:38.495+0000 7fcbc8338640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:38.496 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:38.495+0000 7fcbc8338640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:27:38.496 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:38.495+0000 7fcbc8338640 1 -- 172.21.15.83:0/4235838608 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcbc013dd60 con 0x7fcbc01430c0 2024-03-25T19:27:38.520 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:38.519+0000 7fcbb6ffd640 1 -- 172.21.15.83:0/4235838608 <== mon.1 v1:172.21.15.103:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7fcbb80524c0 con 0x7fcbc01430c0 2024-03-25T19:27:38.520 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:38.519+0000 7fcbb6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:38.520 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:38.519+0000 7fcbb6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:27:38.520 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:27:38.520 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:27:38.523 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:38.522+0000 7fcbc8338640 10 librados: watch_flush enter 2024-03-25T19:27:38.524 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:38.522+0000 7fcbc8338640 10 librados: watch_flush exit 2024-03-25T19:27:38.524 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:38.522+0000 7fcbc8338640 20 client.4591.objecter shutdown clearing up homeless session... 2024-03-25T19:27:38.524 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:38.522+0000 7fcbc8338640 10 client.4591.objecter successfully canceled tick 2024-03-25T19:27:38.524 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:38.522+0000 7fcbc8338640 1 -- 172.21.15.83:0/4235838608 >> v1:172.21.15.103:6800/345072327 conn(0x7fcba406de20 legacy=0x7fcba40702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:38.524 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:38.522+0000 7fcbc8338640 10 monclient: shutdown 2024-03-25T19:27:38.524 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:38.522+0000 7fcbc8338640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:38.524 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:38.522+0000 7fcbc8338640 1 -- 172.21.15.83:0/4235838608 >> v1:172.21.15.103:6789/0 conn(0x7fcbc01430c0 legacy=0x7fcbc0132320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:38.524 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:38.522+0000 7fcbc8338640 1 -- 172.21.15.83:0/4235838608 shutdown_connections 2024-03-25T19:27:38.524 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:38.522+0000 7fcbc8338640 1 -- 172.21.15.83:0/4235838608 >> 172.21.15.83:0/4235838608 conn(0x7fcbc00b7ea0 msgr2=0x7fcbc0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:38.524 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:38.522+0000 7fcbc8338640 1 -- 172.21.15.83:0/4235838608 shutdown_connections 2024-03-25T19:27:38.524 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:38.522+0000 7fcbc8338640 1 -- 172.21.15.83:0/4235838608 wait complete. 2024-03-25T19:27:38.524 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:38.522+0000 7fcbc8338640 1 librados: shutdown 2024-03-25T19:27:38.535 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:27:45.341 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:27:45.455 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.455+0000 7fa9c520d640 10 monclient: get_monmap_and_config 2024-03-25T19:27:45.455 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.455+0000 7fa9c520d640 10 monclient: build_initial_monmap 2024-03-25T19:27:45.455 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.455+0000 7fa9c520d640 10 monclient: monmap: 2024-03-25T19:27:45.455 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:27:45.455 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:27:45.455 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:27:45.456330+0000 2024-03-25T19:27:45.455 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:27:45.456330+0000 2024-03-25T19:27:45.455 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:45.455 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:27:45.456 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:27:45.456 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:27:45.456 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:27:45.456 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:27:45.457 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.456+0000 7fa9c520d640 1 Processor -- start 2024-03-25T19:27:45.457 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.456+0000 7fa9c520d640 1 -- start start 2024-03-25T19:27:45.457 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.456+0000 7fa9c520d640 10 monclient: init 2024-03-25T19:27:45.457 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.457+0000 7fa9c520d640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:45.458 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.457+0000 7fa9c520d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:27:45.458 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.457+0000 7fa9c520d640 10 monclient(hunting): picked mon.noname-a con 0x7fa9c00b4d60 addr v1:172.21.15.83:6789/0 2024-03-25T19:27:45.458 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.457+0000 7fa9c520d640 10 monclient(hunting): picked mon.noname-c con 0x7fa9c00b1730 addr v1:172.21.15.103:6789/0 2024-03-25T19:27:45.458 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.457+0000 7fa9c520d640 10 monclient(hunting): picked mon.noname-b con 0x7fa9c014be90 addr v1:172.21.15.83:6790/0 2024-03-25T19:27:45.458 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.457+0000 7fa9c520d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9c00af940 con 0x7fa9c00b4d60 2024-03-25T19:27:45.458 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.457+0000 7fa9c520d640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9c00b0a70 con 0x7fa9c00b1730 2024-03-25T19:27:45.458 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.457+0000 7fa9c520d640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9c00b0d50 con 0x7fa9c014be90 2024-03-25T19:27:45.458 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.457+0000 7fa9c520d640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:45.458 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.457+0000 7fa9c520d640 10 monclient(hunting): authenticate will time out at 6522.729031s 2024-03-25T19:27:45.464 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.464+0000 7fa9bf59e640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fa9c014be90 0x7fa9c014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:33070/0 (socket says 172.21.15.83:33070) 2024-03-25T19:27:45.464 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.464+0000 7fa9bf59e640 1 -- 172.21.15.83:0/3253993777 learned_addr learned my addr 172.21.15.83:0/3253993777 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:27:45.473 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.474+0000 7fa9be59c640 1 -- 172.21.15.83:0/3253993777 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3230309259 0 0) 0x7fa9c00b0d50 con 0x7fa9c014be90 2024-03-25T19:27:45.474 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.474+0000 7fa9be59c640 10 monclient(hunting): _negotiate 2024-03-25T19:27:45.474 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.474+0000 7fa9be59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:45.474 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.474+0000 7fa9be59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:45.474 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.474+0000 7fa9be59c640 10 monclient(hunting): my global_id is 4598 2024-03-25T19:27:45.474 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.474+0000 7fa9be59c640 1 -- 172.21.15.83:0/3253993777 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9ac003520 con 0x7fa9c014be90 2024-03-25T19:27:45.477 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.477+0000 7fa9be59c640 1 -- 172.21.15.83:0/3253993777 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1675002646 0 0) 0x7fa9c00af940 con 0x7fa9c00b4d60 2024-03-25T19:27:45.477 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.478+0000 7fa9be59c640 10 monclient(hunting): _negotiate 2024-03-25T19:27:45.477 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.478+0000 7fa9be59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:45.477 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.478+0000 7fa9be59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:45.477 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.478+0000 7fa9be59c640 10 monclient(hunting): my global_id is 4584 2024-03-25T19:27:45.477 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.478+0000 7fa9be59c640 1 -- 172.21.15.83:0/3253993777 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9c00b0d50 con 0x7fa9c00b4d60 2024-03-25T19:27:45.492 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.492+0000 7fa9be59c640 1 -- 172.21.15.83:0/3253993777 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3264003639 0 0) 0x7fa9ac003520 con 0x7fa9c014be90 2024-03-25T19:27:45.493 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.492+0000 7fa9be59c640 1 -- 172.21.15.83:0/3253993777 >> v1:172.21.15.103:6789/0 conn(0x7fa9c00b1730 legacy=0x7fa9c00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:45.493 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.492+0000 7fa9be59c640 1 -- 172.21.15.83:0/3253993777 >> v1:172.21.15.83:6789/0 conn(0x7fa9c00b4d60 legacy=0x7fa9c013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:45.493 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.492+0000 7fa9be59c640 10 monclient: _finish_hunting 0 2024-03-25T19:27:45.493 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.492+0000 7fa9be59c640 1 monclient: found mon.noname-b 2024-03-25T19:27:45.493 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.492+0000 7fa9be59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:45.493 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.492+0000 7fa9be59c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.83:6790/0 2024-03-25T19:27:45.493 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.493+0000 7fa9be59c640 1 -- 172.21.15.83:0/3253993777 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c00b1030 con 0x7fa9c014be90 2024-03-25T19:27:45.493 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.493+0000 7fa9be59c640 10 monclient: _finish_auth 0 2024-03-25T19:27:45.493 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.493+0000 7fa9be59c640 10 monclient: _check_auth_tickets 2024-03-25T19:27:45.493 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.493+0000 7fa9be59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:45.493 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.493+0000 7fa9be59c640 1 -- 172.21.15.83:0/3253993777 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fa9b8001a10 con 0x7fa9c014be90 2024-03-25T19:27:45.493 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.493+0000 7fa9be59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:45.493 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.493+0000 7fa9be59c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:27:45.493 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.493+0000 7fa9be59c640 10 monclient: dump: 2024-03-25T19:27:45.493 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:27:45.494 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:27:45.494 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:45.494 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:45.494 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:45.494 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:27:45.494 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:27:45.494 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:27:45.494 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:27:45.494 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:27:45.494 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.493+0000 7fa9c520d640 5 monclient: authenticate success, global_id 4598 2024-03-25T19:27:45.494 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.493+0000 7fa9c520d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:27:45.509 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.510+0000 7fa9be59c640 1 -- 172.21.15.83:0/3253993777 <== mon.2 v1:172.21.15.83:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9b8001b70 con 0x7fa9c014be90 2024-03-25T19:27:45.510 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.510+0000 7fa9be59c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:45.510 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.510+0000 7fa9be59c640 1 -- 172.21.15.83:0/3253993777 <== mon.2 v1:172.21.15.83:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fa9b8001f80 con 0x7fa9c014be90 2024-03-25T19:27:45.510 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.510+0000 7fa9be59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:45.510 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.510+0000 7fa9be59c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:27:45.510 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.510+0000 7fa9be59c640 10 monclient: dump: 2024-03-25T19:27:45.510 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:27:45.510 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:27:45.510 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:45.510 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:45.510 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:45.510 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:27:45.510 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:27:45.510 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:27:45.511 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:27:45.511 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:27:45.511 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.510+0000 7fa9c520d640 10 monclient: get_monmap_and_config success 2024-03-25T19:27:45.511 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.510+0000 7fa9c520d640 4 set_mon_vals no callback set 2024-03-25T19:27:45.513 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.511+0000 7fa9c520d640 10 monclient: shutdown 2024-03-25T19:27:45.513 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.511+0000 7fa9c520d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:45.513 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.511+0000 7fa9c520d640 1 -- 172.21.15.83:0/3253993777 >> v1:172.21.15.83:6790/0 conn(0x7fa9c014be90 legacy=0x7fa9c014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:45.513 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.511+0000 7fa9c520d640 1 -- 172.21.15.83:0/3253993777 shutdown_connections 2024-03-25T19:27:45.513 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.511+0000 7fa9c520d640 1 -- 172.21.15.83:0/3253993777 >> 172.21.15.83:0/3253993777 conn(0x7fa9c00617a0 msgr2=0x7fa9c0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:45.513 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.511+0000 7fa9c520d640 1 -- 172.21.15.83:0/3253993777 shutdown_connections 2024-03-25T19:27:45.513 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.511+0000 7fa9c520d640 1 -- 172.21.15.83:0/3253993777 wait complete. 2024-03-25T19:27:45.513 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.512+0000 7fa9c520d640 10 monclient: build_initial_monmap 2024-03-25T19:27:45.514 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.512+0000 7fa9c520d640 10 monclient: monmap: 2024-03-25T19:27:45.514 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:27:45.514 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:27:45.514 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:27:45.514 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:27:45.514 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:45.514 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:27:45.514 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:27:45.514 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:27:45.514 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:27:45.514 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:27:45.514 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.512+0000 7fa9c520d640 1 librados: starting msgr at 2024-03-25T19:27:45.514 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.512+0000 7fa9c520d640 1 librados: starting objecter 2024-03-25T19:27:45.514 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.512+0000 7fa9c520d640 1 Processor -- start 2024-03-25T19:27:45.515 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.512+0000 7fa9c520d640 1 -- start start 2024-03-25T19:27:45.515 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.512+0000 7fa9c520d640 1 librados: setting wanted keys 2024-03-25T19:27:45.515 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.512+0000 7fa9c520d640 1 librados: calling monclient init 2024-03-25T19:27:45.515 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.512+0000 7fa9c520d640 10 monclient: init 2024-03-25T19:27:45.515 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.513+0000 7fa9c520d640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:45.515 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.513+0000 7fa9c520d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:27:45.515 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.513+0000 7fa9c520d640 10 monclient(hunting): picked mon.noname-c con 0x7fa9c00b1730 addr v1:172.21.15.83:6790/0 2024-03-25T19:27:45.515 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.513+0000 7fa9c520d640 10 monclient(hunting): picked mon.noname-b con 0x7fa9c00b4d60 addr v1:172.21.15.103:6789/0 2024-03-25T19:27:45.515 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.513+0000 7fa9c520d640 10 monclient(hunting): picked mon.noname-a con 0x7fa9c014be90 addr v1:172.21.15.83:6789/0 2024-03-25T19:27:45.515 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.513+0000 7fa9c520d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9c0138490 con 0x7fa9c014be90 2024-03-25T19:27:45.515 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.513+0000 7fa9c520d640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9c0136cc0 con 0x7fa9c00b4d60 2024-03-25T19:27:45.515 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.513+0000 7fa9c520d640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9c0140520 con 0x7fa9c00b1730 2024-03-25T19:27:45.515 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.513+0000 7fa9c520d640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:45.515 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.513+0000 7fa9c520d640 10 monclient(hunting): authenticate will time out at 6522.785368s 2024-03-25T19:27:45.517 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.517+0000 7fa9bf59e640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fa9c014be90 0x7fa9c01e2480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:35504/0 (socket says 172.21.15.83:35504) 2024-03-25T19:27:45.517 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.517+0000 7fa9bf59e640 1 -- 172.21.15.83:0/2198813097 learned_addr learned my addr 172.21.15.83:0/2198813097 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:27:45.520 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.521+0000 7fa9b77fe640 10 client.?.objecter ms_handle_connect 0x7fa9c014be90 2024-03-25T19:27:45.520 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.521+0000 7fa9b77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:45.524 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.525+0000 7fa9b77fe640 10 client.?.objecter ms_handle_connect 0x7fa9c00b1730 2024-03-25T19:27:45.524 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.525+0000 7fa9b77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:45.533 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.534+0000 7fa9b77fe640 1 -- 172.21.15.83:0/2198813097 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4242862736 0 0) 0x7fa9c0138490 con 0x7fa9c014be90 2024-03-25T19:27:45.533 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.534+0000 7fa9b77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:27:45.534 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.534+0000 7fa9b77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:45.534 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.534+0000 7fa9b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:45.534 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.534+0000 7fa9b77fe640 10 monclient(hunting): my global_id is 4587 2024-03-25T19:27:45.534 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.534+0000 7fa9b77fe640 1 -- 172.21.15.83:0/2198813097 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9a0004eb0 con 0x7fa9c014be90 2024-03-25T19:27:45.535 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.535+0000 7fa9b77fe640 10 client.?.objecter ms_handle_connect 0x7fa9c00b4d60 2024-03-25T19:27:45.535 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.535+0000 7fa9b77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:45.536 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.536+0000 7fa9b77fe640 1 -- 172.21.15.83:0/2198813097 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1111880288 0 0) 0x7fa9c0140520 con 0x7fa9c00b1730 2024-03-25T19:27:45.536 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.536+0000 7fa9b77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:27:45.536 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.536+0000 7fa9b77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:45.536 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.536+0000 7fa9b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:45.536 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.536+0000 7fa9b77fe640 10 monclient(hunting): my global_id is 4601 2024-03-25T19:27:45.536 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.536+0000 7fa9b77fe640 1 -- 172.21.15.83:0/2198813097 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9c0138490 con 0x7fa9c00b1730 2024-03-25T19:27:45.552 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.552+0000 7fa9b77fe640 1 -- 172.21.15.83:0/2198813097 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2150111695 0 0) 0x7fa9a0004eb0 con 0x7fa9c014be90 2024-03-25T19:27:45.552 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.552+0000 7fa9b77fe640 1 -- 172.21.15.83:0/2198813097 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9c0140520 con 0x7fa9c014be90 2024-03-25T19:27:45.553 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.552+0000 7fa9b77fe640 1 -- 172.21.15.83:0/2198813097 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fa9b8001ec0 con 0x7fa9c014be90 2024-03-25T19:27:45.553 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.552+0000 7fa9b77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:45.553 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.552+0000 7fa9b77fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:27:45.553 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.552+0000 7fa9b77fe640 10 monclient(hunting): dump: 2024-03-25T19:27:45.553 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:27:45.553 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:27:45.553 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:45.553 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:45.553 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:45.553 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:27:45.553 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:27:45.553 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:27:45.553 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:27:45.553 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:27:45.553 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.552+0000 7fa9b77fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:27:45.554 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.553+0000 7fa9b77fe640 1 -- 172.21.15.83:0/2198813097 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3248408525 0 0) 0x7fa9c0136cc0 con 0x7fa9c00b4d60 2024-03-25T19:27:45.554 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.553+0000 7fa9b77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:27:45.554 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.553+0000 7fa9b77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:45.554 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.553+0000 7fa9b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:45.554 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.553+0000 7fa9b77fe640 10 monclient(hunting): my global_id is 4597 2024-03-25T19:27:45.554 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.553+0000 7fa9b77fe640 1 -- 172.21.15.83:0/2198813097 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9a0004eb0 con 0x7fa9c00b4d60 2024-03-25T19:27:45.560 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.560+0000 7fa9b77fe640 1 -- 172.21.15.83:0/2198813097 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1734067087 0 0) 0x7fa9c0138490 con 0x7fa9c00b1730 2024-03-25T19:27:45.560 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.560+0000 7fa9b77fe640 1 -- 172.21.15.83:0/2198813097 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9c0136cc0 con 0x7fa9c00b1730 2024-03-25T19:27:45.560 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.560+0000 7fa9b77fe640 1 -- 172.21.15.83:0/2198813097 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fa9b00018b0 con 0x7fa9c00b1730 2024-03-25T19:27:45.560 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.560+0000 7fa9b77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:45.560 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.560+0000 7fa9b77fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:27:45.560 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.560+0000 7fa9b77fe640 10 monclient(hunting): dump: 2024-03-25T19:27:45.560 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:27:45.560 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:27:45.561 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:45.561 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:45.561 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:45.561 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:27:45.561 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:27:45.561 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:27:45.561 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:27:45.561 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:27:45.568 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.568+0000 7fa9b77fe640 1 -- 172.21.15.83:0/2198813097 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1812348246 0 0) 0x7fa9c0140520 con 0x7fa9c014be90 2024-03-25T19:27:45.568 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.568+0000 7fa9b77fe640 1 -- 172.21.15.83:0/2198813097 >> v1:172.21.15.83:6790/0 conn(0x7fa9c00b1730 legacy=0x7fa9c0140940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:45.568 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.568+0000 7fa9b77fe640 1 -- 172.21.15.83:0/2198813097 >> v1:172.21.15.103:6789/0 conn(0x7fa9c00b4d60 legacy=0x7fa9c0144e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:45.568 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.568+0000 7fa9b77fe640 10 monclient: _finish_hunting 0 2024-03-25T19:27:45.568 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.568+0000 7fa9b77fe640 1 monclient: found mon.a 2024-03-25T19:27:45.568 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.568+0000 7fa9b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:45.569 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.568+0000 7fa9b77fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:27:45.569 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.568+0000 7fa9b77fe640 1 -- 172.21.15.83:0/2198813097 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c0138700 con 0x7fa9c014be90 2024-03-25T19:27:45.569 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.568+0000 7fa9b77fe640 10 monclient: _finish_auth 0 2024-03-25T19:27:45.569 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.568+0000 7fa9b77fe640 10 monclient: _check_auth_tickets 2024-03-25T19:27:45.569 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.568+0000 7fa9b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:45.569 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.568+0000 7fa9c520d640 5 monclient: authenticate success, global_id 4587 2024-03-25T19:27:45.569 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.568+0000 7fa9c520d640 10 monclient: _renew_subs 2024-03-25T19:27:45.569 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.568+0000 7fa9c520d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:27:45.569 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.568+0000 7fa9c520d640 1 -- 172.21.15.83:0/2198813097 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9c0136660 con 0x7fa9c014be90 2024-03-25T19:27:45.569 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.568+0000 7fa9c520d640 10 client.4587.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:27:45.569 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.568+0000 7fa9c520d640 10 monclient: _renew_subs 2024-03-25T19:27:45.569 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.568+0000 7fa9c520d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:27:45.569 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.568+0000 7fa9c520d640 1 -- 172.21.15.83:0/2198813097 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9c0137350 con 0x7fa9c014be90 2024-03-25T19:27:45.569 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.568+0000 7fa9c520d640 1 librados: init done 2024-03-25T19:27:45.570 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.569+0000 7fa9c520d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:45.570 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.569+0000 7fa9c520d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:45.570 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.569+0000 7fa9c520d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:27:45.570 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.569+0000 7fa9c520d640 1 -- 172.21.15.83:0/2198813097 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9c00627b0 con 0x7fa9c014be90 2024-03-25T19:27:45.877 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.877+0000 7fa9b77fe640 1 -- 172.21.15.83:0/2198813097 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9b8004e10 con 0x7fa9c014be90 2024-03-25T19:27:45.877 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.877+0000 7fa9b77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:45.877 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.877+0000 7fa9b77fe640 1 -- 172.21.15.83:0/2198813097 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fa9b8002100 con 0x7fa9c014be90 2024-03-25T19:27:45.877 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.877+0000 7fa9b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:45.877 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.877+0000 7fa9b77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:27:45.877 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.877+0000 7fa9b77fe640 10 monclient: dump: 2024-03-25T19:27:45.877 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:27:45.877 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:27:45.877 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:45.877 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:45.878 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:45.878 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:27:45.878 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:27:45.878 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:27:45.878 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:27:45.878 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:27:45.878 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.877+0000 7fa9bdd9b640 4 set_mon_vals no callback set 2024-03-25T19:27:45.879 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.878+0000 7fa9b77fe640 1 -- 172.21.15.83:0/2198813097 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7fa9b801afd0 con 0x7fa9c014be90 2024-03-25T19:27:45.879 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.878+0000 7fa9bed9d640 10 monclient: get_auth_request con 0x7fa9a006df70 auth_method 0 2024-03-25T19:27:45.879 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.879+0000 7fa9b77fe640 10 client.4587.objecter ms_handle_connect 0x7fa9a006df70 2024-03-25T19:27:45.884 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.884+0000 7fa9b77fe640 1 -- 172.21.15.83:0/2198813097 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7fa9b8054c30 con 0x7fa9c014be90 2024-03-25T19:27:45.884 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.884+0000 7fa9b77fe640 10 client.4587.objecter ms_dispatch 0x7fa9c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:27:45.884 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.884+0000 7fa9b77fe640 3 client.4587.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:27:45.884 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.884+0000 7fa9b77fe640 3 client.4587.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:27:45.884 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.884+0000 7fa9b77fe640 20 client.4587.objecter dump_active .. 0 homeless 2024-03-25T19:27:45.884 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.884+0000 7fa9b77fe640 1 -- 172.21.15.83:0/2198813097 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa9b8050570 con 0x7fa9c014be90 2024-03-25T19:27:45.884 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.884+0000 7fa9b77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:45.884 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:45.884+0000 7fa9b77fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:27:46.139 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:46.139+0000 7fa9c520d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:46.139 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:46.139+0000 7fa9c520d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:46.139 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:46.139+0000 7fa9c520d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:27:46.139 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:46.139+0000 7fa9c520d640 1 -- 172.21.15.83:0/2198813097 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa9c013c450 con 0x7fa9c014be90 2024-03-25T19:27:46.178 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:46.178+0000 7fa9b77fe640 1 -- 172.21.15.83:0/2198813097 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7fa9b8052670 con 0x7fa9c014be90 2024-03-25T19:27:46.178 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:46.178+0000 7fa9b77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:46.178 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:46.178+0000 7fa9b77fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:27:46.179 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:27:46.179 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:27:46.181 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:46.181+0000 7fa9c520d640 10 librados: watch_flush enter 2024-03-25T19:27:46.181 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:46.181+0000 7fa9c520d640 10 librados: watch_flush exit 2024-03-25T19:27:46.181 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:46.181+0000 7fa9c520d640 20 client.4587.objecter shutdown clearing up homeless session... 2024-03-25T19:27:46.181 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:46.181+0000 7fa9c520d640 10 client.4587.objecter successfully canceled tick 2024-03-25T19:27:46.181 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:46.181+0000 7fa9c520d640 1 -- 172.21.15.83:0/2198813097 >> v1:172.21.15.103:6800/345072327 conn(0x7fa9a006df70 legacy=0x7fa9a0070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:46.181 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:46.181+0000 7fa9c520d640 10 monclient: shutdown 2024-03-25T19:27:46.182 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:46.181+0000 7fa9c520d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:46.182 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:46.181+0000 7fa9c520d640 1 -- 172.21.15.83:0/2198813097 >> v1:172.21.15.83:6789/0 conn(0x7fa9c014be90 legacy=0x7fa9c01e2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:46.182 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:46.181+0000 7fa9c520d640 1 -- 172.21.15.83:0/2198813097 shutdown_connections 2024-03-25T19:27:46.182 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:46.181+0000 7fa9c520d640 1 -- 172.21.15.83:0/2198813097 >> 172.21.15.83:0/2198813097 conn(0x7fa9c00617a0 msgr2=0x7fa9c00b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:46.182 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:46.181+0000 7fa9c520d640 1 -- 172.21.15.83:0/2198813097 shutdown_connections 2024-03-25T19:27:46.182 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:46.181+0000 7fa9c520d640 1 -- 172.21.15.83:0/2198813097 wait complete. 2024-03-25T19:27:46.182 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:46.181+0000 7fa9c520d640 1 librados: shutdown 2024-03-25T19:27:46.195 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:27:48.764 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:08:30.691 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:48.811 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:08:30.739 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:48.845 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:08:30.771 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:48.873 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:08:30.800 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:48.876 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:08:30.786 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:48.877 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:08:30.804 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:48.883 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:08:30.793 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:48.916 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:08:30.844 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:48.923 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:08:30.834 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:53.001 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:27:53.113 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.113+0000 7f9e8280f640 10 monclient: get_monmap_and_config 2024-03-25T19:27:53.113 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.113+0000 7f9e8280f640 10 monclient: build_initial_monmap 2024-03-25T19:27:53.113 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.113+0000 7f9e8280f640 10 monclient: monmap: 2024-03-25T19:27:53.114 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:27:53.114 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:27:53.114 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:27:53.114731+0000 2024-03-25T19:27:53.114 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:27:53.114731+0000 2024-03-25T19:27:53.114 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:53.114 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:27:53.114 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:27:53.114 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:27:53.114 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:27:53.114 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:27:53.115 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.114+0000 7f9e8280f640 1 Processor -- start 2024-03-25T19:27:53.116 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.114+0000 7f9e8280f640 1 -- start start 2024-03-25T19:27:53.116 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.114+0000 7f9e8280f640 10 monclient: init 2024-03-25T19:27:53.116 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.115+0000 7f9e8280f640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:53.116 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.115+0000 7f9e8280f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:27:53.116 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.115+0000 7f9e8280f640 10 monclient(hunting): picked mon.noname-a con 0x7f9e7c0b4d60 addr v1:172.21.15.83:6789/0 2024-03-25T19:27:53.116 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.115+0000 7f9e8280f640 10 monclient(hunting): picked mon.noname-c con 0x7f9e7c0b1730 addr v1:172.21.15.103:6789/0 2024-03-25T19:27:53.116 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.115+0000 7f9e8280f640 10 monclient(hunting): picked mon.noname-b con 0x7f9e7c14be90 addr v1:172.21.15.83:6790/0 2024-03-25T19:27:53.116 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.115+0000 7f9e8280f640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e7c0af940 con 0x7f9e7c0b4d60 2024-03-25T19:27:53.116 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.115+0000 7f9e8280f640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e7c0b0a70 con 0x7f9e7c0b1730 2024-03-25T19:27:53.116 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.115+0000 7f9e8280f640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e7c0b0d50 con 0x7f9e7c14be90 2024-03-25T19:27:53.116 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.115+0000 7f9e8280f640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:53.116 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.115+0000 7f9e8280f640 10 monclient(hunting): authenticate will time out at 6530.386928s 2024-03-25T19:27:53.116 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.115+0000 7f9e80dac640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f9e7c14be90 0x7f9e7c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:48814/0 (socket says 172.21.15.83:48814) 2024-03-25T19:27:53.116 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.115+0000 7f9e80dac640 1 -- 172.21.15.83:0/2760052975 learned_addr learned my addr 172.21.15.83:0/2760052975 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:27:53.168 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.169+0000 7f9e7affd640 1 -- 172.21.15.83:0/2760052975 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 946287397 0 0) 0x7f9e7c0af940 con 0x7f9e7c0b4d60 2024-03-25T19:27:53.169 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.169+0000 7f9e7affd640 10 monclient(hunting): _negotiate 2024-03-25T19:27:53.169 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.169+0000 7f9e7affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:53.169 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.169+0000 7f9e7affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:53.169 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.169+0000 7f9e7affd640 10 monclient(hunting): my global_id is 4590 2024-03-25T19:27:53.169 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.169+0000 7f9e7affd640 1 -- 172.21.15.83:0/2760052975 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e64003520 con 0x7f9e7c0b4d60 2024-03-25T19:27:53.191 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.191+0000 7f9e7affd640 1 -- 172.21.15.83:0/2760052975 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 393576013 0 0) 0x7f9e64003520 con 0x7f9e7c0b4d60 2024-03-25T19:27:53.191 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.191+0000 7f9e7affd640 1 -- 172.21.15.83:0/2760052975 >> v1:172.21.15.83:6790/0 conn(0x7f9e7c14be90 legacy=0x7f9e7c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:53.191 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.191+0000 7f9e7affd640 1 -- 172.21.15.83:0/2760052975 >> v1:172.21.15.103:6789/0 conn(0x7f9e7c0b1730 legacy=0x7f9e7c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:53.191 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.191+0000 7f9e7affd640 10 monclient: _finish_hunting 0 2024-03-25T19:27:53.191 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.191+0000 7f9e7affd640 1 monclient: found mon.noname-a 2024-03-25T19:27:53.191 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.191+0000 7f9e7affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:53.191 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.191+0000 7f9e7affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.83:6789/0 2024-03-25T19:27:53.192 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.191+0000 7f9e7affd640 1 -- 172.21.15.83:0/2760052975 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e7c0b1030 con 0x7f9e7c0b4d60 2024-03-25T19:27:53.192 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.191+0000 7f9e7affd640 10 monclient: _finish_auth 0 2024-03-25T19:27:53.192 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.191+0000 7f9e7affd640 10 monclient: _check_auth_tickets 2024-03-25T19:27:53.192 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.191+0000 7f9e7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:53.192 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.191+0000 7f9e7affd640 1 -- 172.21.15.83:0/2760052975 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f9e70001690 con 0x7f9e7c0b4d60 2024-03-25T19:27:53.192 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.191+0000 7f9e7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:53.192 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.191+0000 7f9e7affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:27:53.192 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.191+0000 7f9e7affd640 10 monclient: dump: 2024-03-25T19:27:53.192 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:27:53.192 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:27:53.192 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:53.192 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:53.192 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:53.192 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:27:53.192 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:27:53.192 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:27:53.193 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:27:53.193 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:27:53.193 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.191+0000 7f9e8280f640 5 monclient: authenticate success, global_id 4590 2024-03-25T19:27:53.193 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.191+0000 7f9e8280f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:27:53.204 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.204+0000 7f9e7affd640 1 -- 172.21.15.83:0/2760052975 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9e700017f0 con 0x7f9e7c0b4d60 2024-03-25T19:27:53.204 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.204+0000 7f9e7affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:53.205 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.205+0000 7f9e7affd640 1 -- 172.21.15.83:0/2760052975 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f9e70001c40 con 0x7f9e7c0b4d60 2024-03-25T19:27:53.205 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.205+0000 7f9e7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:53.205 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.205+0000 7f9e7affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:27:53.205 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.205+0000 7f9e7affd640 10 monclient: dump: 2024-03-25T19:27:53.205 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:27:53.205 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:27:53.205 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:53.205 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:53.205 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:53.205 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:27:53.205 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:27:53.205 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:27:53.205 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:27:53.205 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:27:53.206 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.205+0000 7f9e8280f640 10 monclient: get_monmap_and_config success 2024-03-25T19:27:53.206 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.205+0000 7f9e8280f640 4 set_mon_vals no callback set 2024-03-25T19:27:53.208 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.205+0000 7f9e8280f640 10 monclient: shutdown 2024-03-25T19:27:53.208 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.205+0000 7f9e8280f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:53.208 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.205+0000 7f9e8280f640 1 -- 172.21.15.83:0/2760052975 >> v1:172.21.15.83:6789/0 conn(0x7f9e7c0b4d60 legacy=0x7f9e7c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:53.208 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.206+0000 7f9e8280f640 1 -- 172.21.15.83:0/2760052975 shutdown_connections 2024-03-25T19:27:53.208 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.206+0000 7f9e8280f640 1 -- 172.21.15.83:0/2760052975 >> 172.21.15.83:0/2760052975 conn(0x7f9e7c0617a0 msgr2=0x7f9e7c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:53.208 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.206+0000 7f9e8280f640 1 -- 172.21.15.83:0/2760052975 shutdown_connections 2024-03-25T19:27:53.208 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.206+0000 7f9e8280f640 1 -- 172.21.15.83:0/2760052975 wait complete. 2024-03-25T19:27:53.208 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.206+0000 7f9e8280f640 10 monclient: build_initial_monmap 2024-03-25T19:27:53.208 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.206+0000 7f9e8280f640 10 monclient: monmap: 2024-03-25T19:27:53.208 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:27:53.208 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:27:53.208 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:27:53.208 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:27:53.209 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:53.209 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:27:53.209 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:27:53.209 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:27:53.209 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:27:53.209 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:27:53.209 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.207+0000 7f9e8280f640 1 librados: starting msgr at 2024-03-25T19:27:53.209 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.207+0000 7f9e8280f640 1 librados: starting objecter 2024-03-25T19:27:53.209 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.207+0000 7f9e8280f640 1 Processor -- start 2024-03-25T19:27:53.209 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.207+0000 7f9e8280f640 1 -- start start 2024-03-25T19:27:53.209 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.207+0000 7f9e8280f640 1 librados: setting wanted keys 2024-03-25T19:27:53.209 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.207+0000 7f9e8280f640 1 librados: calling monclient init 2024-03-25T19:27:53.209 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.207+0000 7f9e8280f640 10 monclient: init 2024-03-25T19:27:53.209 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.208+0000 7f9e8280f640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:53.210 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.208+0000 7f9e8280f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:27:53.210 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.208+0000 7f9e8280f640 10 monclient(hunting): picked mon.noname-c con 0x7f9e7c0b1730 addr v1:172.21.15.83:6790/0 2024-03-25T19:27:53.210 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.208+0000 7f9e8280f640 10 monclient(hunting): picked mon.noname-b con 0x7f9e7c0b4d60 addr v1:172.21.15.103:6789/0 2024-03-25T19:27:53.210 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.208+0000 7f9e8280f640 10 monclient(hunting): picked mon.noname-a con 0x7f9e7c14be90 addr v1:172.21.15.83:6789/0 2024-03-25T19:27:53.210 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.208+0000 7f9e8280f640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e7c142f70 con 0x7f9e7c14be90 2024-03-25T19:27:53.210 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.208+0000 7f9e8280f640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e7c1411b0 con 0x7f9e7c0b4d60 2024-03-25T19:27:53.210 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.208+0000 7f9e8280f640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e7c143250 con 0x7f9e7c0b1730 2024-03-25T19:27:53.210 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.208+0000 7f9e8280f640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:53.210 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.208+0000 7f9e8280f640 10 monclient(hunting): authenticate will time out at 6530.480136s 2024-03-25T19:27:53.212 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.212+0000 7f9e7b7fe640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f9e7c0b4d60 0x7f9e7c13d0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:46158/0 (socket says 172.21.15.83:46158) 2024-03-25T19:27:53.212 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.212+0000 7f9e7b7fe640 1 -- 172.21.15.83:0/2493223651 learned_addr learned my addr 172.21.15.83:0/2493223651 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:27:53.212 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.212+0000 7f9e78ff9640 10 client.?.objecter ms_handle_connect 0x7f9e7c0b4d60 2024-03-25T19:27:53.212 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.212+0000 7f9e78ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:53.218 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.218+0000 7f9e78ff9640 1 -- 172.21.15.83:0/2493223651 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2159083118 0 0) 0x7f9e7c1411b0 con 0x7f9e7c0b4d60 2024-03-25T19:27:53.218 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.218+0000 7f9e78ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:27:53.218 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.218+0000 7f9e78ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:53.218 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.218+0000 7f9e78ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:53.218 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.218+0000 7f9e78ff9640 10 monclient(hunting): my global_id is 4603 2024-03-25T19:27:53.218 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.218+0000 7f9e78ff9640 1 -- 172.21.15.83:0/2493223651 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e68004eb0 con 0x7f9e7c0b4d60 2024-03-25T19:27:53.221 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.221+0000 7f9e78ff9640 10 client.?.objecter ms_handle_connect 0x7f9e7c14be90 2024-03-25T19:27:53.221 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.221+0000 7f9e78ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:53.225 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.225+0000 7f9e78ff9640 1 -- 172.21.15.83:0/2493223651 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2991642292 0 0) 0x7f9e68004eb0 con 0x7f9e7c0b4d60 2024-03-25T19:27:53.225 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.226+0000 7f9e78ff9640 1 -- 172.21.15.83:0/2493223651 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e7c1411b0 con 0x7f9e7c0b4d60 2024-03-25T19:27:53.225 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.226+0000 7f9e78ff9640 1 -- 172.21.15.83:0/2493223651 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f9e6c0019d0 con 0x7f9e7c0b4d60 2024-03-25T19:27:53.226 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.226+0000 7f9e78ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:53.226 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.226+0000 7f9e78ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:27:53.226 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.226+0000 7f9e78ff9640 10 monclient(hunting): dump: 2024-03-25T19:27:53.226 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:27:53.226 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:27:53.226 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:53.226 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:53.226 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:53.226 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:27:53.226 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:27:53.226 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:27:53.226 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:27:53.226 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:27:53.226 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.226+0000 7f9e78ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:27:53.234 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.234+0000 7f9e78ff9640 10 client.?.objecter ms_handle_connect 0x7f9e7c0b1730 2024-03-25T19:27:53.234 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.234+0000 7f9e78ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:53.236 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.236+0000 7f9e78ff9640 1 -- 172.21.15.83:0/2493223651 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3097864564 0 0) 0x7f9e7c142f70 con 0x7f9e7c14be90 2024-03-25T19:27:53.236 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.236+0000 7f9e78ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:27:53.236 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.236+0000 7f9e78ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:53.236 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.236+0000 7f9e78ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:53.237 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.236+0000 7f9e78ff9640 10 monclient(hunting): my global_id is 4593 2024-03-25T19:27:53.237 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.236+0000 7f9e78ff9640 1 -- 172.21.15.83:0/2493223651 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e68004eb0 con 0x7f9e7c14be90 2024-03-25T19:27:53.240 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.239+0000 7f9e78ff9640 1 -- 172.21.15.83:0/2493223651 <== mon.1 v1:172.21.15.103:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1438563168 0 0) 0x7f9e7c1411b0 con 0x7f9e7c0b4d60 2024-03-25T19:27:53.240 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.240+0000 7f9e78ff9640 1 -- 172.21.15.83:0/2493223651 >> v1:172.21.15.83:6790/0 conn(0x7f9e7c0b1730 legacy=0x7f9e7c1372b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:53.240 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.240+0000 7f9e78ff9640 1 -- 172.21.15.83:0/2493223651 >> v1:172.21.15.83:6789/0 conn(0x7f9e7c14be90 legacy=0x7f9e7c141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:53.240 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.240+0000 7f9e78ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:27:53.240 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.240+0000 7f9e78ff9640 1 monclient: found mon.b 2024-03-25T19:27:53.240 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.240+0000 7f9e78ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:53.240 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.240+0000 7f9e78ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:27:53.240 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.240+0000 7f9e78ff9640 1 -- 172.21.15.83:0/2493223651 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e7c140c90 con 0x7f9e7c0b4d60 2024-03-25T19:27:53.240 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.240+0000 7f9e78ff9640 10 monclient: _finish_auth 0 2024-03-25T19:27:53.241 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.240+0000 7f9e78ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:27:53.241 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.240+0000 7f9e78ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:53.241 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.240+0000 7f9e8280f640 5 monclient: authenticate success, global_id 4603 2024-03-25T19:27:53.241 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.240+0000 7f9e8280f640 10 monclient: _renew_subs 2024-03-25T19:27:53.241 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.240+0000 7f9e8280f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:27:53.241 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.240+0000 7f9e8280f640 1 -- 172.21.15.83:0/2493223651 --> v1:172.21.15.103:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e7c1379c0 con 0x7f9e7c0b4d60 2024-03-25T19:27:53.241 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.240+0000 7f9e8280f640 10 client.4603.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:27:53.241 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.240+0000 7f9e8280f640 10 monclient: _renew_subs 2024-03-25T19:27:53.241 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.240+0000 7f9e8280f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:27:53.241 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.240+0000 7f9e8280f640 1 -- 172.21.15.83:0/2493223651 --> v1:172.21.15.103:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e7c13d9e0 con 0x7f9e7c0b4d60 2024-03-25T19:27:53.241 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.240+0000 7f9e8280f640 1 librados: init done 2024-03-25T19:27:53.242 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.241+0000 7f9e8280f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:53.242 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.241+0000 7f9e8280f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:53.242 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.241+0000 7f9e8280f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:27:53.242 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.241+0000 7f9e8280f640 1 -- 172.21.15.83:0/2493223651 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e7c0627b0 con 0x7f9e7c0b4d60 2024-03-25T19:27:53.459 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.459+0000 7f9e78ff9640 1 -- 172.21.15.83:0/2493223651 <== mon.1 v1:172.21.15.103:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9e6c001c90 con 0x7f9e7c0b4d60 2024-03-25T19:27:53.459 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.459+0000 7f9e78ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:53.460 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.459+0000 7f9e7a7fc640 4 set_mon_vals no callback set 2024-03-25T19:27:53.522 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.522+0000 7f9e78ff9640 1 -- 172.21.15.83:0/2493223651 <== mon.1 v1:172.21.15.103:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f9e6c003a60 con 0x7f9e7c0b4d60 2024-03-25T19:27:53.522 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.522+0000 7f9e78ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:53.522 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.522+0000 7f9e78ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:27:53.522 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.522+0000 7f9e78ff9640 10 monclient: dump: 2024-03-25T19:27:53.522 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:27:53.522 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:27:53.522 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:53.522 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:27:53.522 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:53.522 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:27:53.522 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:27:53.522 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:27:53.523 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:27:53.523 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:27:53.523 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.523+0000 7f9e78ff9640 1 -- 172.21.15.83:0/2493223651 <== mon.1 v1:172.21.15.103:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7f9e6c003c40 con 0x7f9e7c0b4d60 2024-03-25T19:27:53.523 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.523+0000 7f9e7bfff640 10 monclient: get_auth_request con 0x7f9e6806dd40 auth_method 0 2024-03-25T19:27:53.523 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.524+0000 7f9e78ff9640 10 client.4603.objecter ms_handle_connect 0x7f9e6806dd40 2024-03-25T19:27:53.531 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.532+0000 7f9e78ff9640 1 -- 172.21.15.83:0/2493223651 <== mon.1 v1:172.21.15.103:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7f9e6c0548b0 con 0x7f9e7c0b4d60 2024-03-25T19:27:53.531 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.532+0000 7f9e78ff9640 10 client.4603.objecter ms_dispatch 0x7f9e7c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:27:53.532 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.532+0000 7f9e78ff9640 3 client.4603.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:27:53.532 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.532+0000 7f9e78ff9640 3 client.4603.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:27:53.532 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.532+0000 7f9e78ff9640 20 client.4603.objecter dump_active .. 0 homeless 2024-03-25T19:27:53.538 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.538+0000 7f9e78ff9640 1 -- 172.21.15.83:0/2493223651 <== mon.1 v1:172.21.15.103:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9e6c0503c0 con 0x7f9e7c0b4d60 2024-03-25T19:27:53.538 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.538+0000 7f9e78ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:53.538 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.538+0000 7f9e78ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:27:53.793 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.793+0000 7f9e8280f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:53.793 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.793+0000 7f9e8280f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:53.793 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.793+0000 7f9e8280f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:27:53.793 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.793+0000 7f9e8280f640 1 -- 172.21.15.83:0/2493223651 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9e7c140e20 con 0x7f9e7c0b4d60 2024-03-25T19:27:53.819 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.819+0000 7f9e78ff9640 1 -- 172.21.15.83:0/2493223651 <== mon.1 v1:172.21.15.103:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7f9e6c0522f0 con 0x7f9e7c0b4d60 2024-03-25T19:27:53.819 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.819+0000 7f9e78ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:53.819 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.819+0000 7f9e78ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:27:53.819 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:27:53.819 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:27:53.822 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.822+0000 7f9e8280f640 10 librados: watch_flush enter 2024-03-25T19:27:53.822 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.822+0000 7f9e8280f640 10 librados: watch_flush exit 2024-03-25T19:27:53.822 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.822+0000 7f9e8280f640 20 client.4603.objecter shutdown clearing up homeless session... 2024-03-25T19:27:53.822 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.822+0000 7f9e8280f640 10 client.4603.objecter successfully canceled tick 2024-03-25T19:27:53.822 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.822+0000 7f9e8280f640 1 -- 172.21.15.83:0/2493223651 >> v1:172.21.15.103:6800/345072327 conn(0x7f9e6806dd40 legacy=0x7f9e68070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:53.822 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.822+0000 7f9e8280f640 10 monclient: shutdown 2024-03-25T19:27:53.823 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.822+0000 7f9e8280f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:53.823 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.822+0000 7f9e8280f640 1 -- 172.21.15.83:0/2493223651 >> v1:172.21.15.103:6789/0 conn(0x7f9e7c0b4d60 legacy=0x7f9e7c13d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:53.823 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.822+0000 7f9e8280f640 1 -- 172.21.15.83:0/2493223651 shutdown_connections 2024-03-25T19:27:53.823 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.822+0000 7f9e8280f640 1 -- 172.21.15.83:0/2493223651 >> 172.21.15.83:0/2493223651 conn(0x7f9e7c0617a0 msgr2=0x7f9e7c0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:53.823 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.822+0000 7f9e8280f640 1 -- 172.21.15.83:0/2493223651 shutdown_connections 2024-03-25T19:27:53.823 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.822+0000 7f9e8280f640 1 -- 172.21.15.83:0/2493223651 wait complete. 2024-03-25T19:27:53.823 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:27:53.822+0000 7f9e8280f640 1 librados: shutdown 2024-03-25T19:27:53.834 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:28:00.640 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:28:00.757 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.757+0000 7f4fed7cf640 10 monclient: get_monmap_and_config 2024-03-25T19:28:00.757 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.757+0000 7f4fed7cf640 10 monclient: build_initial_monmap 2024-03-25T19:28:00.757 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.757+0000 7f4fed7cf640 10 monclient: monmap: 2024-03-25T19:28:00.757 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:28:00.757 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:28:00.757 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:28:00.758182+0000 2024-03-25T19:28:00.757 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:28:00.758182+0000 2024-03-25T19:28:00.757 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:00.757 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:28:00.757 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:28:00.757 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:28:00.757 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:28:00.758 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:00.759 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.758+0000 7f4fed7cf640 1 Processor -- start 2024-03-25T19:28:00.759 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.758+0000 7f4fed7cf640 1 -- start start 2024-03-25T19:28:00.759 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.758+0000 7f4fed7cf640 10 monclient: init 2024-03-25T19:28:00.759 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.758+0000 7f4fed7cf640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:00.759 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.758+0000 7f4fed7cf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:28:00.759 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.758+0000 7f4fed7cf640 10 monclient(hunting): picked mon.noname-c con 0x7f4fe8061830 addr v1:172.21.15.103:6789/0 2024-03-25T19:28:00.759 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.758+0000 7f4fed7cf640 10 monclient(hunting): picked mon.noname-b con 0x7f4fe813fcd0 addr v1:172.21.15.83:6790/0 2024-03-25T19:28:00.759 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.759+0000 7f4fed7cf640 10 monclient(hunting): picked mon.noname-a con 0x7f4fe80ae840 addr v1:172.21.15.83:6789/0 2024-03-25T19:28:00.759 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.759+0000 7f4fed7cf640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fe806f760 con 0x7f4fe80ae840 2024-03-25T19:28:00.760 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.759+0000 7f4fed7cf640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fe806fa40 con 0x7f4fe8061830 2024-03-25T19:28:00.760 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.759+0000 7f4fed7cf640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fe80ae0c0 con 0x7f4fe813fcd0 2024-03-25T19:28:00.760 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.759+0000 7f4fed7cf640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:00.760 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.759+0000 7f4fed7cf640 10 monclient(hunting): authenticate will time out at 6538.030896s 2024-03-25T19:28:00.764 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.764+0000 7f4fe77fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f4fe80ae840 0x7f4fe8145f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:46228/0 (socket says 172.21.15.83:46228) 2024-03-25T19:28:00.764 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.764+0000 7f4fe77fe640 1 -- 172.21.15.83:0/2441483291 learned_addr learned my addr 172.21.15.83:0/2441483291 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:28:00.779 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.779+0000 7f4fe5ffb640 1 -- 172.21.15.83:0/2441483291 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3381471639 0 0) 0x7f4fe806f760 con 0x7f4fe80ae840 2024-03-25T19:28:00.779 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.779+0000 7f4fe5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:28:00.779 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.779+0000 7f4fe5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:00.779 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.779+0000 7f4fe5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:00.779 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.779+0000 7f4fe5ffb640 10 monclient(hunting): my global_id is 4596 2024-03-25T19:28:00.779 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.779+0000 7f4fe5ffb640 1 -- 172.21.15.83:0/2441483291 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fd0003520 con 0x7f4fe80ae840 2024-03-25T19:28:00.784 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.784+0000 7f4fe5ffb640 1 -- 172.21.15.83:0/2441483291 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3146386264 0 0) 0x7f4fe80ae0c0 con 0x7f4fe813fcd0 2024-03-25T19:28:00.784 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.784+0000 7f4fe5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:28:00.784 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.784+0000 7f4fe5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:00.784 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.784+0000 7f4fe5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:00.785 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.784+0000 7f4fe5ffb640 10 monclient(hunting): my global_id is 4610 2024-03-25T19:28:00.785 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.784+0000 7f4fe5ffb640 1 -- 172.21.15.83:0/2441483291 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fe806f760 con 0x7f4fe813fcd0 2024-03-25T19:28:00.785 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.784+0000 7f4fe5ffb640 1 -- 172.21.15.83:0/2441483291 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2801348178 0 0) 0x7f4fe806fa40 con 0x7f4fe8061830 2024-03-25T19:28:00.785 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.784+0000 7f4fe5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:28:00.785 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.784+0000 7f4fe5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:00.785 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.784+0000 7f4fe5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:00.785 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.784+0000 7f4fe5ffb640 10 monclient(hunting): my global_id is 4606 2024-03-25T19:28:00.785 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.784+0000 7f4fe5ffb640 1 -- 172.21.15.83:0/2441483291 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fe80ae0c0 con 0x7f4fe8061830 2024-03-25T19:28:00.791 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.791+0000 7f4fe5ffb640 1 -- 172.21.15.83:0/2441483291 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 418343209 0 0) 0x7f4fd0003520 con 0x7f4fe80ae840 2024-03-25T19:28:00.791 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.791+0000 7f4fe5ffb640 1 -- 172.21.15.83:0/2441483291 >> v1:172.21.15.83:6790/0 conn(0x7f4fe813fcd0 legacy=0x7f4fe8145810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:00.791 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.791+0000 7f4fe5ffb640 1 -- 172.21.15.83:0/2441483291 >> v1:172.21.15.103:6789/0 conn(0x7f4fe8061830 legacy=0x7f4fe813f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:00.791 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.791+0000 7f4fe5ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:28:00.791 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.791+0000 7f4fe5ffb640 1 monclient: found mon.noname-a 2024-03-25T19:28:00.792 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.791+0000 7f4fe5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:00.792 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.791+0000 7f4fe5ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.83:6789/0 2024-03-25T19:28:00.792 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.791+0000 7f4fe5ffb640 1 -- 172.21.15.83:0/2441483291 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fe8073040 con 0x7f4fe80ae840 2024-03-25T19:28:00.792 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.791+0000 7f4fe5ffb640 10 monclient: _finish_auth 0 2024-03-25T19:28:00.792 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.791+0000 7f4fe5ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:28:00.792 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.791+0000 7f4fe5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:00.792 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.791+0000 7f4fe5ffb640 1 -- 172.21.15.83:0/2441483291 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f4fe0001a10 con 0x7f4fe80ae840 2024-03-25T19:28:00.792 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.791+0000 7f4fe5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:00.792 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.791+0000 7f4fe5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:28:00.792 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.791+0000 7f4fe5ffb640 10 monclient: dump: 2024-03-25T19:28:00.792 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:28:00.792 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:28:00.792 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:00.792 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:00.792 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:00.792 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:28:00.792 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:28:00.793 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:28:00.793 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:28:00.793 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:00.793 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.791+0000 7f4fed7cf640 5 monclient: authenticate success, global_id 4596 2024-03-25T19:28:00.793 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.791+0000 7f4fed7cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:00.805 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.805+0000 7f4fe5ffb640 1 -- 172.21.15.83:0/2441483291 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4fe0001b70 con 0x7f4fe80ae840 2024-03-25T19:28:00.805 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.805+0000 7f4fe5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:00.805 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.805+0000 7f4fe5ffb640 1 -- 172.21.15.83:0/2441483291 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f4fe0001f80 con 0x7f4fe80ae840 2024-03-25T19:28:00.805 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.805+0000 7f4fed7cf640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:00.805 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.805+0000 7f4fed7cf640 4 set_mon_vals no callback set 2024-03-25T19:28:00.806 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.805+0000 7f4fe5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:00.806 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.805+0000 7f4fe5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:00.806 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.805+0000 7f4fe5ffb640 10 monclient: dump: 2024-03-25T19:28:00.806 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:28:00.806 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:28:00.806 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:00.806 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:00.806 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:00.806 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:28:00.806 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:28:00.806 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:28:00.806 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:28:00.806 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:00.809 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.806+0000 7f4fed7cf640 10 monclient: shutdown 2024-03-25T19:28:00.809 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.806+0000 7f4fed7cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:00.809 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.806+0000 7f4fed7cf640 1 -- 172.21.15.83:0/2441483291 >> v1:172.21.15.83:6789/0 conn(0x7f4fe80ae840 legacy=0x7f4fe8145f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:00.809 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.806+0000 7f4fed7cf640 1 -- 172.21.15.83:0/2441483291 shutdown_connections 2024-03-25T19:28:00.809 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.806+0000 7f4fed7cf640 1 -- 172.21.15.83:0/2441483291 >> 172.21.15.83:0/2441483291 conn(0x7f4fe80b0ef0 msgr2=0x7f4fe80b12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:00.809 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.806+0000 7f4fed7cf640 1 -- 172.21.15.83:0/2441483291 shutdown_connections 2024-03-25T19:28:00.809 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.807+0000 7f4fed7cf640 1 -- 172.21.15.83:0/2441483291 wait complete. 2024-03-25T19:28:00.809 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.807+0000 7f4fed7cf640 10 monclient: build_initial_monmap 2024-03-25T19:28:00.809 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.807+0000 7f4fed7cf640 10 monclient: monmap: 2024-03-25T19:28:00.809 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:28:00.810 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:00.810 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:28:00.810 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:28:00.810 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:00.810 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:28:00.810 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:28:00.810 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:28:00.810 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:28:00.810 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:00.810 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.808+0000 7f4fed7cf640 1 librados: starting msgr at 2024-03-25T19:28:00.810 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.808+0000 7f4fed7cf640 1 librados: starting objecter 2024-03-25T19:28:00.810 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.808+0000 7f4fed7cf640 1 Processor -- start 2024-03-25T19:28:00.810 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.808+0000 7f4fed7cf640 1 -- start start 2024-03-25T19:28:00.810 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.808+0000 7f4fed7cf640 1 librados: setting wanted keys 2024-03-25T19:28:00.811 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.808+0000 7f4fed7cf640 1 librados: calling monclient init 2024-03-25T19:28:00.811 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.808+0000 7f4fed7cf640 10 monclient: init 2024-03-25T19:28:00.811 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.808+0000 7f4fed7cf640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:00.811 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.808+0000 7f4fed7cf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:28:00.811 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.808+0000 7f4fed7cf640 10 monclient(hunting): picked mon.noname-b con 0x7f4fe8061830 addr v1:172.21.15.103:6789/0 2024-03-25T19:28:00.811 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.809+0000 7f4fed7cf640 10 monclient(hunting): picked mon.noname-a con 0x7f4fe80ae840 addr v1:172.21.15.83:6789/0 2024-03-25T19:28:00.811 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.809+0000 7f4fed7cf640 10 monclient(hunting): picked mon.noname-c con 0x7f4fe813fcd0 addr v1:172.21.15.83:6790/0 2024-03-25T19:28:00.811 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.809+0000 7f4fed7cf640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fe813eb00 con 0x7f4fe80ae840 2024-03-25T19:28:00.811 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.809+0000 7f4fed7cf640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fe8138ae0 con 0x7f4fe8061830 2024-03-25T19:28:00.811 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.809+0000 7f4fed7cf640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fe8135cd0 con 0x7f4fe813fcd0 2024-03-25T19:28:00.811 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.809+0000 7f4fed7cf640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:00.811 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.809+0000 7f4fed7cf640 10 monclient(hunting): authenticate will time out at 6538.081181s 2024-03-25T19:28:00.812 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.810+0000 7f4fe67fc640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f4fe80ae840 0x7f4fe8137bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:46280/0 (socket says 172.21.15.83:46280) 2024-03-25T19:28:00.812 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.810+0000 7f4fe67fc640 1 -- 172.21.15.83:0/373215156 learned_addr learned my addr 172.21.15.83:0/373215156 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:28:00.812 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.811+0000 7f4fcf7fe640 10 client.?.objecter ms_handle_connect 0x7f4fe80ae840 2024-03-25T19:28:00.812 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.811+0000 7f4fcf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:00.815 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.815+0000 7f4fcf7fe640 10 client.?.objecter ms_handle_connect 0x7f4fe813fcd0 2024-03-25T19:28:00.815 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.815+0000 7f4fcf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:00.820 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.820+0000 7f4fcf7fe640 1 -- 172.21.15.83:0/373215156 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 561602357 0 0) 0x7f4fe806f760 con 0x7f4fe80ae840 2024-03-25T19:28:00.820 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.820+0000 7f4fcf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:28:00.820 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.820+0000 7f4fcf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:00.820 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.820+0000 7f4fcf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:00.820 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.820+0000 7f4fcf7fe640 10 monclient(hunting): my global_id is 4599 2024-03-25T19:28:00.820 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.820+0000 7f4fcf7fe640 1 -- 172.21.15.83:0/373215156 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fd4004eb0 con 0x7f4fe80ae840 2024-03-25T19:28:00.820 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.820+0000 7f4fcf7fe640 10 client.?.objecter ms_handle_connect 0x7f4fe8061830 2024-03-25T19:28:00.820 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.820+0000 7f4fcf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:00.826 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.827+0000 7f4fcf7fe640 1 -- 172.21.15.83:0/373215156 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 839266846 0 0) 0x7f4fe8138ae0 con 0x7f4fe8061830 2024-03-25T19:28:00.827 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.827+0000 7f4fcf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:28:00.827 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.827+0000 7f4fcf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:00.827 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.827+0000 7f4fcf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:00.827 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.827+0000 7f4fcf7fe640 10 monclient(hunting): my global_id is 4609 2024-03-25T19:28:00.827 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.827+0000 7f4fcf7fe640 1 -- 172.21.15.83:0/373215156 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fe806f760 con 0x7f4fe8061830 2024-03-25T19:28:00.832 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.833+0000 7f4fcf7fe640 1 -- 172.21.15.83:0/373215156 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2032362303 0 0) 0x7f4fe8135cd0 con 0x7f4fe813fcd0 2024-03-25T19:28:00.833 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.833+0000 7f4fcf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:28:00.833 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.833+0000 7f4fcf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:00.833 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.833+0000 7f4fcf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:00.833 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.833+0000 7f4fcf7fe640 10 monclient(hunting): my global_id is 4613 2024-03-25T19:28:00.833 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.833+0000 7f4fcf7fe640 1 -- 172.21.15.83:0/373215156 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fe8138ae0 con 0x7f4fe813fcd0 2024-03-25T19:28:00.842 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.842+0000 7f4fcf7fe640 1 -- 172.21.15.83:0/373215156 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 957205127 0 0) 0x7f4fe806f760 con 0x7f4fe8061830 2024-03-25T19:28:00.842 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.842+0000 7f4fcf7fe640 1 -- 172.21.15.83:0/373215156 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4fe8135cd0 con 0x7f4fe8061830 2024-03-25T19:28:00.842 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.842+0000 7f4fcf7fe640 1 -- 172.21.15.83:0/373215156 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f4fd8001810 con 0x7f4fe8061830 2024-03-25T19:28:00.842 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.842+0000 7f4fcf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:00.842 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.842+0000 7f4fcf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:28:00.842 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.842+0000 7f4fcf7fe640 10 monclient(hunting): dump: 2024-03-25T19:28:00.842 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:28:00.842 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:28:00.842 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:00.842 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:00.842 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:00.843 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:28:00.843 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:28:00.843 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:28:00.843 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:28:00.843 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:00.843 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.842+0000 7f4fcf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:00.847 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.847+0000 7f4fcf7fe640 1 -- 172.21.15.83:0/373215156 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 730865210 0 0) 0x7f4fd4004eb0 con 0x7f4fe80ae840 2024-03-25T19:28:00.847 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.847+0000 7f4fcf7fe640 1 -- 172.21.15.83:0/373215156 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4fe806f760 con 0x7f4fe80ae840 2024-03-25T19:28:00.847 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.847+0000 7f4fcf7fe640 1 -- 172.21.15.83:0/373215156 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f4fdc001870 con 0x7f4fe80ae840 2024-03-25T19:28:00.847 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.847+0000 7f4fcf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:00.847 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.847+0000 7f4fcf7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:00.847 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.847+0000 7f4fcf7fe640 10 monclient(hunting): dump: 2024-03-25T19:28:00.847 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:28:00.847 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:28:00.847 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:00.848 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:00.848 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:00.848 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:28:00.848 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:28:00.848 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:28:00.848 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:28:00.848 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:00.852 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.853+0000 7f4fcf7fe640 1 -- 172.21.15.83:0/373215156 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3220256821 0 0) 0x7f4fe8138ae0 con 0x7f4fe813fcd0 2024-03-25T19:28:00.852 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.853+0000 7f4fcf7fe640 1 -- 172.21.15.83:0/373215156 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4fd4004eb0 con 0x7f4fe813fcd0 2024-03-25T19:28:00.852 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.853+0000 7f4fcf7fe640 1 -- 172.21.15.83:0/373215156 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f4fe0001e80 con 0x7f4fe813fcd0 2024-03-25T19:28:00.853 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.853+0000 7f4fcf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:00.853 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.853+0000 7f4fcf7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:28:00.853 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.853+0000 7f4fcf7fe640 10 monclient(hunting): dump: 2024-03-25T19:28:00.853 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:28:00.853 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:28:00.853 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:00.853 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:00.853 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:00.853 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:28:00.853 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:28:00.853 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:28:00.853 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:28:00.853 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:00.855 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.854+0000 7f4fcf7fe640 1 -- 172.21.15.83:0/373215156 <== mon.1 v1:172.21.15.103:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 154238183 0 0) 0x7f4fe8135cd0 con 0x7f4fe8061830 2024-03-25T19:28:00.855 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.854+0000 7f4fcf7fe640 1 -- 172.21.15.83:0/373215156 >> v1:172.21.15.83:6790/0 conn(0x7f4fe813fcd0 legacy=0x7f4fe81e2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:00.855 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.854+0000 7f4fcf7fe640 1 -- 172.21.15.83:0/373215156 >> v1:172.21.15.83:6789/0 conn(0x7f4fe80ae840 legacy=0x7f4fe8137bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:00.855 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.854+0000 7f4fcf7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:28:00.855 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.854+0000 7f4fcf7fe640 1 monclient: found mon.b 2024-03-25T19:28:00.855 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.854+0000 7f4fcf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:00.855 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.854+0000 7f4fcf7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:28:00.855 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.854+0000 7f4fcf7fe640 1 -- 172.21.15.83:0/373215156 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fe8138d30 con 0x7f4fe8061830 2024-03-25T19:28:00.856 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.854+0000 7f4fcf7fe640 10 monclient: _finish_auth 0 2024-03-25T19:28:00.856 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.854+0000 7f4fcf7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:28:00.856 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.854+0000 7f4fcf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:00.856 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.854+0000 7f4fed7cf640 5 monclient: authenticate success, global_id 4609 2024-03-25T19:28:00.856 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.854+0000 7f4fed7cf640 10 monclient: _renew_subs 2024-03-25T19:28:00.856 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.854+0000 7f4fed7cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:28:00.856 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.854+0000 7f4fed7cf640 1 -- 172.21.15.83:0/373215156 --> v1:172.21.15.103:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fe81369e0 con 0x7f4fe8061830 2024-03-25T19:28:00.856 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.854+0000 7f4fed7cf640 10 client.4609.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:00.856 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.854+0000 7f4fed7cf640 10 monclient: _renew_subs 2024-03-25T19:28:00.856 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.854+0000 7f4fed7cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:28:00.856 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.854+0000 7f4fed7cf640 1 -- 172.21.15.83:0/373215156 --> v1:172.21.15.103:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fe8137680 con 0x7f4fe8061830 2024-03-25T19:28:00.856 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.854+0000 7f4fed7cf640 1 librados: init done 2024-03-25T19:28:00.856 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.855+0000 7f4fed7cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:00.856 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.855+0000 7f4fed7cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:00.856 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.855+0000 7f4fed7cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:28:00.856 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:00.855+0000 7f4fed7cf640 1 -- 172.21.15.83:0/373215156 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fe8137980 con 0x7f4fe8061830 2024-03-25T19:28:01.134 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:01.134+0000 7f4fcf7fe640 1 -- 172.21.15.83:0/373215156 <== mon.1 v1:172.21.15.103:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4fd8001d20 con 0x7f4fe8061830 2024-03-25T19:28:01.134 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:01.134+0000 7f4fcf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:01.134 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:01.134+0000 7f4fcf7fe640 1 -- 172.21.15.83:0/373215156 <== mon.1 v1:172.21.15.103:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f4fd8003690 con 0x7f4fe8061830 2024-03-25T19:28:01.134 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:01.134+0000 7f4fcf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:01.134 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:01.134+0000 7f4fcf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:01.134 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:01.134+0000 7f4fcf7fe640 10 monclient: dump: 2024-03-25T19:28:01.134 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:28:01.134 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:28:01.134 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:01.134 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:01.135 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:01.135 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:28:01.135 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:28:01.135 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:28:01.135 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:28:01.135 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:01.135 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:01.134+0000 7f4fe57fa640 4 set_mon_vals no callback set 2024-03-25T19:28:01.135 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:01.135+0000 7f4fcf7fe640 1 -- 172.21.15.83:0/373215156 <== mon.1 v1:172.21.15.103:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7f4fd80196a0 con 0x7f4fe8061830 2024-03-25T19:28:01.136 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:01.136+0000 7f4fe67fc640 10 monclient: get_auth_request con 0x7f4fd406e0a0 auth_method 0 2024-03-25T19:28:01.136 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:01.136+0000 7f4fcf7fe640 10 client.4609.objecter ms_handle_connect 0x7f4fd406e0a0 2024-03-25T19:28:01.137 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:01.137+0000 7f4fcf7fe640 1 -- 172.21.15.83:0/373215156 <== mon.1 v1:172.21.15.103:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7f4fd8053320 con 0x7f4fe8061830 2024-03-25T19:28:01.137 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:01.137+0000 7f4fcf7fe640 10 client.4609.objecter ms_dispatch 0x7f4fe8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:28:01.137 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:01.137+0000 7f4fcf7fe640 3 client.4609.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:28:01.137 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:01.137+0000 7f4fcf7fe640 3 client.4609.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:28:01.137 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:01.137+0000 7f4fcf7fe640 20 client.4609.objecter dump_active .. 0 homeless 2024-03-25T19:28:01.139 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:01.140+0000 7f4fcf7fe640 1 -- 172.21.15.83:0/373215156 <== mon.1 v1:172.21.15.103:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4fd804ec60 con 0x7f4fe8061830 2024-03-25T19:28:01.139 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:01.140+0000 7f4fcf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:01.140 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:01.140+0000 7f4fcf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:01.368 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:01.368+0000 7f4fed7cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:01.368 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:01.368+0000 7f4fed7cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:01.368 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:01.368+0000 7f4fed7cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:28:01.368 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:01.368+0000 7f4fed7cf640 1 -- 172.21.15.83:0/373215156 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4fe80627b0 con 0x7f4fe8061830 2024-03-25T19:28:01.533 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:01.533+0000 7f4fcf7fe640 1 -- 172.21.15.83:0/373215156 <== mon.1 v1:172.21.15.103:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7f4fd8050d60 con 0x7f4fe8061830 2024-03-25T19:28:01.533 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:01.533+0000 7f4fcf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:01.533 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:01.533+0000 7f4fcf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:28:01.533 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:28:01.533 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:28:01.535 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:01.535+0000 7f4fed7cf640 10 librados: watch_flush enter 2024-03-25T19:28:01.536 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:01.535+0000 7f4fed7cf640 10 librados: watch_flush exit 2024-03-25T19:28:01.536 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:01.535+0000 7f4fed7cf640 20 client.4609.objecter shutdown clearing up homeless session... 2024-03-25T19:28:01.536 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:01.535+0000 7f4fed7cf640 10 client.4609.objecter successfully canceled tick 2024-03-25T19:28:01.536 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:01.535+0000 7f4fed7cf640 1 -- 172.21.15.83:0/373215156 >> v1:172.21.15.103:6800/345072327 conn(0x7f4fd406e0a0 legacy=0x7f4fd4070560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:01.536 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:01.535+0000 7f4fed7cf640 10 monclient: shutdown 2024-03-25T19:28:01.536 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:01.535+0000 7f4fed7cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:01.536 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:01.535+0000 7f4fed7cf640 1 -- 172.21.15.83:0/373215156 >> v1:172.21.15.103:6789/0 conn(0x7f4fe8061830 legacy=0x7f4fe8139690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:01.536 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:01.535+0000 7f4fed7cf640 1 -- 172.21.15.83:0/373215156 shutdown_connections 2024-03-25T19:28:01.536 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:01.535+0000 7f4fed7cf640 1 -- 172.21.15.83:0/373215156 >> 172.21.15.83:0/373215156 conn(0x7f4fe80b0ef0 msgr2=0x7f4fe80aecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:01.536 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:01.535+0000 7f4fed7cf640 1 -- 172.21.15.83:0/373215156 shutdown_connections 2024-03-25T19:28:01.536 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:01.535+0000 7f4fed7cf640 1 -- 172.21.15.83:0/373215156 wait complete. 2024-03-25T19:28:01.536 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:01.535+0000 7f4fed7cf640 1 librados: shutdown 2024-03-25T19:28:01.548 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:28:08.355 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:28:08.472 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.473+0000 7f210a72b640 10 monclient: get_monmap_and_config 2024-03-25T19:28:08.472 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.473+0000 7f210a72b640 10 monclient: build_initial_monmap 2024-03-25T19:28:08.473 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.473+0000 7f210a72b640 10 monclient: monmap: 2024-03-25T19:28:08.473 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:28:08.473 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:28:08.473 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:28:08.473919+0000 2024-03-25T19:28:08.473 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:28:08.473919+0000 2024-03-25T19:28:08.473 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:08.473 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:28:08.473 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:28:08.473 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:28:08.473 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:28:08.473 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:08.475 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.473+0000 7f210a72b640 1 Processor -- start 2024-03-25T19:28:08.475 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.473+0000 7f210a72b640 1 -- start start 2024-03-25T19:28:08.475 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.473+0000 7f210a72b640 10 monclient: init 2024-03-25T19:28:08.475 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.474+0000 7f210a72b640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:08.475 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.474+0000 7f210a72b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:28:08.475 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.474+0000 7f210a72b640 10 monclient(hunting): picked mon.noname-a con 0x7f21040a98f0 addr v1:172.21.15.83:6789/0 2024-03-25T19:28:08.475 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.474+0000 7f210a72b640 10 monclient(hunting): picked mon.noname-b con 0x7f210413f710 addr v1:172.21.15.83:6790/0 2024-03-25T19:28:08.475 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.474+0000 7f210a72b640 10 monclient(hunting): picked mon.noname-c con 0x7f21041430c0 addr v1:172.21.15.103:6789/0 2024-03-25T19:28:08.475 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.474+0000 7f210a72b640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2104142520 con 0x7f21040a98f0 2024-03-25T19:28:08.475 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.474+0000 7f210a72b640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21040b28a0 con 0x7f21041430c0 2024-03-25T19:28:08.475 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.474+0000 7f210a72b640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21040b2a30 con 0x7f210413f710 2024-03-25T19:28:08.475 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.474+0000 7f210a72b640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:08.475 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.474+0000 7f210a72b640 10 monclient(hunting): authenticate will time out at 6545.746206s 2024-03-25T19:28:08.480 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.481+0000 7f2103fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f21040a98f0 0x7f21040a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:34774/0 (socket says 172.21.15.83:34774) 2024-03-25T19:28:08.480 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.481+0000 7f2103fff640 1 -- 172.21.15.83:0/3144160647 learned_addr learned my addr 172.21.15.83:0/3144160647 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:28:08.491 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.492+0000 7f2102ffd640 1 -- 172.21.15.83:0/3144160647 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1443284046 0 0) 0x7f2104142520 con 0x7f21040a98f0 2024-03-25T19:28:08.492 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.492+0000 7f2102ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:08.492 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.492+0000 7f2102ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:08.492 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.492+0000 7f2102ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:08.492 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.492+0000 7f2102ffd640 10 monclient(hunting): my global_id is 4602 2024-03-25T19:28:08.492 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.492+0000 7f2102ffd640 1 -- 172.21.15.83:0/3144160647 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20ec003520 con 0x7f21040a98f0 2024-03-25T19:28:08.494 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.495+0000 7f2102ffd640 1 -- 172.21.15.83:0/3144160647 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2321387220 0 0) 0x7f21040b28a0 con 0x7f21041430c0 2024-03-25T19:28:08.494 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.495+0000 7f2102ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:08.494 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.495+0000 7f2102ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:08.494 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.495+0000 7f2102ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:08.494 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.495+0000 7f2102ffd640 10 monclient(hunting): my global_id is 4612 2024-03-25T19:28:08.494 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.495+0000 7f2102ffd640 1 -- 172.21.15.83:0/3144160647 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2104142520 con 0x7f21041430c0 2024-03-25T19:28:08.497 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.497+0000 7f2102ffd640 1 -- 172.21.15.83:0/3144160647 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4155335409 0 0) 0x7f21040b2a30 con 0x7f210413f710 2024-03-25T19:28:08.497 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.497+0000 7f2102ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:08.497 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.497+0000 7f2102ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:08.497 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.497+0000 7f2102ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:08.497 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.497+0000 7f2102ffd640 10 monclient(hunting): my global_id is 4616 2024-03-25T19:28:08.498 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.497+0000 7f2102ffd640 1 -- 172.21.15.83:0/3144160647 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21040b28a0 con 0x7f210413f710 2024-03-25T19:28:08.499 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.498+0000 7f2102ffd640 1 -- 172.21.15.83:0/3144160647 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2254169792 0 0) 0x7f20ec003520 con 0x7f21040a98f0 2024-03-25T19:28:08.499 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.498+0000 7f2102ffd640 1 -- 172.21.15.83:0/3144160647 >> v1:172.21.15.83:6790/0 conn(0x7f210413f710 legacy=0x7f21041429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:08.499 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.498+0000 7f2102ffd640 1 -- 172.21.15.83:0/3144160647 >> v1:172.21.15.103:6789/0 conn(0x7f21041430c0 legacy=0x7f210414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:08.499 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.498+0000 7f2102ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:28:08.499 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.498+0000 7f2102ffd640 1 monclient: found mon.noname-a 2024-03-25T19:28:08.499 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.498+0000 7f2102ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:08.499 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.498+0000 7f2102ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.83:6789/0 2024-03-25T19:28:08.499 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.498+0000 7f2102ffd640 1 -- 172.21.15.83:0/3144160647 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21040b44c0 con 0x7f21040a98f0 2024-03-25T19:28:08.499 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.498+0000 7f2102ffd640 10 monclient: _finish_auth 0 2024-03-25T19:28:08.499 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.498+0000 7f2102ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:28:08.499 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.498+0000 7f2102ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:08.499 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.498+0000 7f2102ffd640 1 -- 172.21.15.83:0/3144160647 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f20f4001690 con 0x7f21040a98f0 2024-03-25T19:28:08.499 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.498+0000 7f210a72b640 5 monclient: authenticate success, global_id 4602 2024-03-25T19:28:08.500 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.498+0000 7f210a72b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:08.500 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.498+0000 7f2102ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:08.500 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.498+0000 7f2102ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:28:08.500 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.498+0000 7f2102ffd640 10 monclient: dump: 2024-03-25T19:28:08.500 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:28:08.500 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:28:08.500 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:08.500 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:08.500 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:08.500 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:28:08.500 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:28:08.500 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:28:08.500 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:28:08.500 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:08.501 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.498+0000 7f210a72b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:08.501 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.502+0000 7f2102ffd640 1 -- 172.21.15.83:0/3144160647 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20f40017f0 con 0x7f21040a98f0 2024-03-25T19:28:08.501 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.502+0000 7f2102ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:08.502 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.502+0000 7f2102ffd640 1 -- 172.21.15.83:0/3144160647 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f20f4001c00 con 0x7f21040a98f0 2024-03-25T19:28:08.502 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.502+0000 7f210a72b640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:08.502 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.502+0000 7f210a72b640 4 set_mon_vals no callback set 2024-03-25T19:28:08.502 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.502+0000 7f2102ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:08.502 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.502+0000 7f2102ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:08.502 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.502+0000 7f2102ffd640 10 monclient: dump: 2024-03-25T19:28:08.502 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:28:08.502 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:28:08.502 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:08.502 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:08.502 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:08.502 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:28:08.502 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:28:08.503 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:28:08.503 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:28:08.503 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:08.504 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.503+0000 7f210a72b640 10 monclient: shutdown 2024-03-25T19:28:08.504 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.503+0000 7f210a72b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:08.504 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.503+0000 7f210a72b640 1 -- 172.21.15.83:0/3144160647 >> v1:172.21.15.83:6789/0 conn(0x7f21040a98f0 legacy=0x7f21040a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:08.504 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.503+0000 7f210a72b640 1 -- 172.21.15.83:0/3144160647 shutdown_connections 2024-03-25T19:28:08.504 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.503+0000 7f210a72b640 1 -- 172.21.15.83:0/3144160647 >> 172.21.15.83:0/3144160647 conn(0x7f21040b7ea0 msgr2=0x7f21040b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:08.504 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.503+0000 7f210a72b640 1 -- 172.21.15.83:0/3144160647 shutdown_connections 2024-03-25T19:28:08.505 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.503+0000 7f210a72b640 1 -- 172.21.15.83:0/3144160647 wait complete. 2024-03-25T19:28:08.505 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.504+0000 7f210a72b640 10 monclient: build_initial_monmap 2024-03-25T19:28:08.505 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.504+0000 7f210a72b640 10 monclient: monmap: 2024-03-25T19:28:08.505 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:28:08.505 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:08.505 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:28:08.505 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:28:08.505 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:08.505 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:28:08.505 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:28:08.505 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:28:08.505 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:28:08.505 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:08.505 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.504+0000 7f210a72b640 1 librados: starting msgr at 2024-03-25T19:28:08.506 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.504+0000 7f210a72b640 1 librados: starting objecter 2024-03-25T19:28:08.506 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.504+0000 7f210a72b640 1 Processor -- start 2024-03-25T19:28:08.506 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.504+0000 7f210a72b640 1 -- start start 2024-03-25T19:28:08.506 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.504+0000 7f210a72b640 1 librados: setting wanted keys 2024-03-25T19:28:08.506 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.504+0000 7f210a72b640 1 librados: calling monclient init 2024-03-25T19:28:08.506 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.504+0000 7f210a72b640 10 monclient: init 2024-03-25T19:28:08.507 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.505+0000 7f210a72b640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:08.507 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.505+0000 7f210a72b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:28:08.507 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.505+0000 7f210a72b640 10 monclient(hunting): picked mon.noname-c con 0x7f21040a98f0 addr v1:172.21.15.83:6790/0 2024-03-25T19:28:08.507 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.505+0000 7f210a72b640 10 monclient(hunting): picked mon.noname-b con 0x7f210413f710 addr v1:172.21.15.103:6789/0 2024-03-25T19:28:08.507 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.505+0000 7f210a72b640 10 monclient(hunting): picked mon.noname-a con 0x7f21041430c0 addr v1:172.21.15.83:6789/0 2024-03-25T19:28:08.507 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.505+0000 7f210a72b640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2104138970 con 0x7f21041430c0 2024-03-25T19:28:08.507 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.505+0000 7f210a72b640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21041d7810 con 0x7f210413f710 2024-03-25T19:28:08.507 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.505+0000 7f210a72b640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2104139990 con 0x7f21040a98f0 2024-03-25T19:28:08.507 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.505+0000 7f210a72b640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:08.507 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.505+0000 7f210a72b640 10 monclient(hunting): authenticate will time out at 6545.777241s 2024-03-25T19:28:08.520 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.521+0000 7f21037fe640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f210413f710 0x7f21041d7d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:45656/0 (socket says 172.21.15.83:45656) 2024-03-25T19:28:08.520 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.521+0000 7f21037fe640 1 -- 172.21.15.83:0/3097880814 learned_addr learned my addr 172.21.15.83:0/3097880814 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:28:08.526 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.527+0000 7f2100ff9640 10 client.?.objecter ms_handle_connect 0x7f21040a98f0 2024-03-25T19:28:08.527 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.527+0000 7f2100ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:08.529 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.529+0000 7f2100ff9640 10 client.?.objecter ms_handle_connect 0x7f210413f710 2024-03-25T19:28:08.529 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.529+0000 7f2100ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:08.532 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.532+0000 7f2100ff9640 10 client.?.objecter ms_handle_connect 0x7f21041430c0 2024-03-25T19:28:08.532 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.532+0000 7f2100ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:08.538 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.539+0000 7f2100ff9640 1 -- 172.21.15.83:0/3097880814 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3369783957 0 0) 0x7f2104139990 con 0x7f21040a98f0 2024-03-25T19:28:08.539 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.539+0000 7f2100ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:28:08.539 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.539+0000 7f2100ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:08.539 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.539+0000 7f2100ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:08.539 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.539+0000 7f2100ff9640 10 monclient(hunting): my global_id is 4619 2024-03-25T19:28:08.539 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.539+0000 7f2100ff9640 1 -- 172.21.15.83:0/3097880814 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20f0004eb0 con 0x7f21040a98f0 2024-03-25T19:28:08.549 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.549+0000 7f2100ff9640 1 -- 172.21.15.83:0/3097880814 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 526484663 0 0) 0x7f2104138970 con 0x7f21041430c0 2024-03-25T19:28:08.549 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.550+0000 7f2100ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:28:08.549 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.550+0000 7f2100ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:08.549 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.550+0000 7f2100ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:08.549 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.550+0000 7f2100ff9640 10 monclient(hunting): my global_id is 4605 2024-03-25T19:28:08.550 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.550+0000 7f2100ff9640 1 -- 172.21.15.83:0/3097880814 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2104139990 con 0x7f21041430c0 2024-03-25T19:28:08.556 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.556+0000 7f2100ff9640 1 -- 172.21.15.83:0/3097880814 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3416986223 0 0) 0x7f21041d7810 con 0x7f210413f710 2024-03-25T19:28:08.556 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.556+0000 7f2100ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:28:08.556 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.556+0000 7f2100ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:08.556 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.556+0000 7f2100ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:08.556 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.556+0000 7f2100ff9640 10 monclient(hunting): my global_id is 4615 2024-03-25T19:28:08.556 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.556+0000 7f2100ff9640 1 -- 172.21.15.83:0/3097880814 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2104138970 con 0x7f210413f710 2024-03-25T19:28:08.567 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.568+0000 7f2100ff9640 1 -- 172.21.15.83:0/3097880814 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1931634014 0 0) 0x7f20f0004eb0 con 0x7f21040a98f0 2024-03-25T19:28:08.568 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.568+0000 7f2100ff9640 1 -- 172.21.15.83:0/3097880814 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21041d7810 con 0x7f21040a98f0 2024-03-25T19:28:08.568 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.568+0000 7f2100ff9640 1 -- 172.21.15.83:0/3097880814 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f20f4001b00 con 0x7f21040a98f0 2024-03-25T19:28:08.568 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.568+0000 7f2100ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:08.568 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.568+0000 7f2100ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:28:08.568 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.568+0000 7f2100ff9640 10 monclient(hunting): dump: 2024-03-25T19:28:08.568 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:28:08.568 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:28:08.568 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:08.568 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:08.568 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:08.568 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:28:08.568 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:28:08.568 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:28:08.568 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:28:08.568 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:08.569 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.568+0000 7f2100ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:08.572 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.572+0000 7f2100ff9640 1 -- 172.21.15.83:0/3097880814 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1227506379 0 0) 0x7f2104138970 con 0x7f210413f710 2024-03-25T19:28:08.572 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.572+0000 7f2100ff9640 1 -- 172.21.15.83:0/3097880814 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f20f0004eb0 con 0x7f210413f710 2024-03-25T19:28:08.572 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.572+0000 7f2100ff9640 1 -- 172.21.15.83:0/3097880814 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f20f80019b0 con 0x7f210413f710 2024-03-25T19:28:08.572 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.572+0000 7f2100ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:08.572 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.572+0000 7f2100ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:08.572 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.572+0000 7f2100ff9640 10 monclient(hunting): dump: 2024-03-25T19:28:08.572 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:28:08.572 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:28:08.572 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:08.572 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:08.572 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:08.572 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:28:08.573 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:28:08.573 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:28:08.573 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:28:08.573 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:08.576 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.576+0000 7f2100ff9640 1 -- 172.21.15.83:0/3097880814 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1606169484 0 0) 0x7f2104139990 con 0x7f21041430c0 2024-03-25T19:28:08.576 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.576+0000 7f2100ff9640 1 -- 172.21.15.83:0/3097880814 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2104138970 con 0x7f21041430c0 2024-03-25T19:28:08.576 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.576+0000 7f2100ff9640 1 -- 172.21.15.83:0/3097880814 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f20fc001b50 con 0x7f21041430c0 2024-03-25T19:28:08.576 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.576+0000 7f2100ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:08.576 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.576+0000 7f2100ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:08.576 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.576+0000 7f2100ff9640 10 monclient(hunting): dump: 2024-03-25T19:28:08.576 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:28:08.576 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:28:08.576 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:08.576 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:08.577 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:08.577 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:28:08.577 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:28:08.577 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:28:08.577 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:28:08.577 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:08.582 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.581+0000 7f2100ff9640 1 -- 172.21.15.83:0/3097880814 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 391176630 0 0) 0x7f21041d7810 con 0x7f21040a98f0 2024-03-25T19:28:08.582 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.581+0000 7f2100ff9640 1 -- 172.21.15.83:0/3097880814 >> v1:172.21.15.103:6789/0 conn(0x7f210413f710 legacy=0x7f21041d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:08.582 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.581+0000 7f2100ff9640 1 -- 172.21.15.83:0/3097880814 >> v1:172.21.15.83:6789/0 conn(0x7f21041430c0 legacy=0x7f21041d4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:08.582 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.581+0000 7f2100ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:28:08.582 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.581+0000 7f2100ff9640 1 monclient: found mon.c 2024-03-25T19:28:08.582 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.581+0000 7f2100ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:08.582 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.581+0000 7f2100ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:28:08.583 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.581+0000 7f2100ff9640 1 -- 172.21.15.83:0/3097880814 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21041d8440 con 0x7f21040a98f0 2024-03-25T19:28:08.583 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.581+0000 7f2100ff9640 10 monclient: _finish_auth 0 2024-03-25T19:28:08.583 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.581+0000 7f2100ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:28:08.583 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.581+0000 7f2100ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:08.583 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.581+0000 7f210a72b640 5 monclient: authenticate success, global_id 4619 2024-03-25T19:28:08.583 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.581+0000 7f210a72b640 10 monclient: _renew_subs 2024-03-25T19:28:08.583 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.581+0000 7f210a72b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:28:08.583 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.581+0000 7f210a72b640 1 -- 172.21.15.83:0/3097880814 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21041d41e0 con 0x7f21040a98f0 2024-03-25T19:28:08.583 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.581+0000 7f210a72b640 10 client.4619.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:08.583 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.581+0000 7f210a72b640 10 monclient: _renew_subs 2024-03-25T19:28:08.583 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.581+0000 7f210a72b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:28:08.583 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.581+0000 7f210a72b640 1 -- 172.21.15.83:0/3097880814 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2104138d80 con 0x7f21040a98f0 2024-03-25T19:28:08.583 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.581+0000 7f210a72b640 1 librados: init done 2024-03-25T19:28:08.583 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.582+0000 7f210a72b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:08.583 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.582+0000 7f210a72b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:08.584 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.582+0000 7f210a72b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:28:08.584 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.582+0000 7f210a72b640 1 -- 172.21.15.83:0/3097880814 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21040627b0 con 0x7f21040a98f0 2024-03-25T19:28:08.889 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.890+0000 7f2100ff9640 1 -- 172.21.15.83:0/3097880814 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20f4004a90 con 0x7f21040a98f0 2024-03-25T19:28:08.890 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.890+0000 7f2100ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:08.890 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.890+0000 7f2100ff9640 1 -- 172.21.15.83:0/3097880814 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f20f4004da0 con 0x7f21040a98f0 2024-03-25T19:28:08.890 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.890+0000 7f2100ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:08.890 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.890+0000 7f2100ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:28:08.890 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.890+0000 7f2100ff9640 10 monclient: dump: 2024-03-25T19:28:08.890 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:28:08.890 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:28:08.890 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:08.890 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:08.890 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:08.890 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:28:08.890 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:28:08.890 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:28:08.891 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:28:08.891 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:08.891 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.890+0000 7f21027fc640 4 set_mon_vals no callback set 2024-03-25T19:28:08.891 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.890+0000 7f2100ff9640 1 -- 172.21.15.83:0/3097880814 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7f20f401ad70 con 0x7f21040a98f0 2024-03-25T19:28:08.891 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.891+0000 7f21037fe640 10 monclient: get_auth_request con 0x7f20f006e0c0 auth_method 0 2024-03-25T19:28:08.891 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.891+0000 7f2100ff9640 10 client.4619.objecter ms_handle_connect 0x7f20f006e0c0 2024-03-25T19:28:08.898 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.898+0000 7f2100ff9640 1 -- 172.21.15.83:0/3097880814 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7f20f4054870 con 0x7f21040a98f0 2024-03-25T19:28:08.898 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.898+0000 7f2100ff9640 10 client.4619.objecter ms_dispatch 0x7f2104000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:28:08.898 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.898+0000 7f2100ff9640 3 client.4619.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:28:08.898 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.898+0000 7f2100ff9640 3 client.4619.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:28:08.898 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.898+0000 7f2100ff9640 20 client.4619.objecter dump_active .. 0 homeless 2024-03-25T19:28:08.898 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.898+0000 7f2100ff9640 1 -- 172.21.15.83:0/3097880814 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f20f4050380 con 0x7f21040a98f0 2024-03-25T19:28:08.898 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.898+0000 7f2100ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:08.898 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:08.898+0000 7f2100ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:08.925 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:08:50.852 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:08.976 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:08:50.904 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:09.006 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:08:50.934 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:09.032 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:08:50.942 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:09.053 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:08:50.963 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:09.055 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:08:50.983 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:09.061 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:08:50.989 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:09.082 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:08:50.992 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:09.090 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:08:51.017 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:09.104 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:09.104+0000 7f210a72b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:09.104 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:09.104+0000 7f210a72b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:09.104 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:09.104+0000 7f210a72b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:28:09.104 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:09.104+0000 7f210a72b640 1 -- 172.21.15.83:0/3097880814 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21040a9d60 con 0x7f21040a98f0 2024-03-25T19:28:09.136 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:09.137+0000 7f2100ff9640 1 -- 172.21.15.83:0/3097880814 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7f20f40522b0 con 0x7f21040a98f0 2024-03-25T19:28:09.137 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:09.137+0000 7f2100ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:09.137 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:09.137+0000 7f2100ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:28:09.137 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:28:09.137 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:28:09.139 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:09.139+0000 7f210a72b640 10 librados: watch_flush enter 2024-03-25T19:28:09.140 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:09.139+0000 7f210a72b640 10 librados: watch_flush exit 2024-03-25T19:28:09.140 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:09.139+0000 7f210a72b640 20 client.4619.objecter shutdown clearing up homeless session... 2024-03-25T19:28:09.140 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:09.139+0000 7f210a72b640 10 client.4619.objecter successfully canceled tick 2024-03-25T19:28:09.140 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:09.139+0000 7f210a72b640 1 -- 172.21.15.83:0/3097880814 >> v1:172.21.15.103:6800/345072327 conn(0x7f20f006e0c0 legacy=0x7f20f0070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:09.140 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:09.139+0000 7f210a72b640 10 monclient: shutdown 2024-03-25T19:28:09.140 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:09.139+0000 7f210a72b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:09.140 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:09.139+0000 7f210a72b640 1 -- 172.21.15.83:0/3097880814 >> v1:172.21.15.83:6790/0 conn(0x7f21040a98f0 legacy=0x7f2104138260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:09.140 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:09.139+0000 7f210a72b640 1 -- 172.21.15.83:0/3097880814 shutdown_connections 2024-03-25T19:28:09.140 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:09.139+0000 7f210a72b640 1 -- 172.21.15.83:0/3097880814 >> 172.21.15.83:0/3097880814 conn(0x7f21040b7ea0 msgr2=0x7f210413c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:09.140 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:09.139+0000 7f210a72b640 1 -- 172.21.15.83:0/3097880814 shutdown_connections 2024-03-25T19:28:09.140 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:09.139+0000 7f210a72b640 1 -- 172.21.15.83:0/3097880814 wait complete. 2024-03-25T19:28:09.140 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:09.139+0000 7f210a72b640 1 librados: shutdown 2024-03-25T19:28:09.150 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:28:15.956 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:28:16.074 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.074+0000 7fc60bbc1640 10 monclient: get_monmap_and_config 2024-03-25T19:28:16.075 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.074+0000 7fc60bbc1640 10 monclient: build_initial_monmap 2024-03-25T19:28:16.075 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.074+0000 7fc60bbc1640 10 monclient: monmap: 2024-03-25T19:28:16.075 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:28:16.075 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:28:16.075 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:28:16.075793+0000 2024-03-25T19:28:16.075 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:28:16.075793+0000 2024-03-25T19:28:16.075 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:16.075 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:28:16.075 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:28:16.075 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:28:16.075 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:28:16.075 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:16.077 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.075+0000 7fc60bbc1640 1 Processor -- start 2024-03-25T19:28:16.077 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.075+0000 7fc60bbc1640 1 -- start start 2024-03-25T19:28:16.077 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.075+0000 7fc60bbc1640 10 monclient: init 2024-03-25T19:28:16.077 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.076+0000 7fc60bbc1640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:16.077 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.076+0000 7fc60bbc1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:28:16.077 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.076+0000 7fc60bbc1640 10 monclient(hunting): picked mon.noname-a con 0x7fc6040a98f0 addr v1:172.21.15.83:6789/0 2024-03-25T19:28:16.077 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.076+0000 7fc60bbc1640 10 monclient(hunting): picked mon.noname-b con 0x7fc60413f710 addr v1:172.21.15.83:6790/0 2024-03-25T19:28:16.077 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.076+0000 7fc60bbc1640 10 monclient(hunting): picked mon.noname-c con 0x7fc6041430c0 addr v1:172.21.15.103:6789/0 2024-03-25T19:28:16.077 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.076+0000 7fc60bbc1640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc604142520 con 0x7fc6040a98f0 2024-03-25T19:28:16.077 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.076+0000 7fc60bbc1640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6040b28a0 con 0x7fc6041430c0 2024-03-25T19:28:16.077 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.076+0000 7fc60bbc1640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6040b2a30 con 0x7fc60413f710 2024-03-25T19:28:16.078 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.076+0000 7fc60bbc1640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:16.078 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.076+0000 7fc60bbc1640 10 monclient(hunting): authenticate will time out at 6553.348415s 2024-03-25T19:28:16.081 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.081+0000 7fc60995d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fc6040a98f0 0x7fc6040a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:44782/0 (socket says 172.21.15.83:44782) 2024-03-25T19:28:16.081 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.081+0000 7fc60995d640 1 -- 172.21.15.83:0/3411840332 learned_addr learned my addr 172.21.15.83:0/3411840332 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:28:16.093 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.093+0000 7fc60895b640 1 -- 172.21.15.83:0/3411840332 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 6679976 0 0) 0x7fc6040b2a30 con 0x7fc60413f710 2024-03-25T19:28:16.093 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.093+0000 7fc60895b640 10 monclient(hunting): _negotiate 2024-03-25T19:28:16.093 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.093+0000 7fc60895b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:16.093 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.093+0000 7fc60895b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:16.093 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.093+0000 7fc60895b640 10 monclient(hunting): my global_id is 4622 2024-03-25T19:28:16.093 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.093+0000 7fc60895b640 1 -- 172.21.15.83:0/3411840332 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5f0003520 con 0x7fc60413f710 2024-03-25T19:28:16.097 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.097+0000 7fc60895b640 1 -- 172.21.15.83:0/3411840332 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3570406417 0 0) 0x7fc604142520 con 0x7fc6040a98f0 2024-03-25T19:28:16.097 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.097+0000 7fc60895b640 10 monclient(hunting): _negotiate 2024-03-25T19:28:16.097 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.097+0000 7fc60895b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:16.097 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.097+0000 7fc60895b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:16.097 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.097+0000 7fc60895b640 10 monclient(hunting): my global_id is 4608 2024-03-25T19:28:16.097 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.097+0000 7fc60895b640 1 -- 172.21.15.83:0/3411840332 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6040b2a30 con 0x7fc6040a98f0 2024-03-25T19:28:16.101 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.102+0000 7fc60895b640 1 -- 172.21.15.83:0/3411840332 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 726987599 0 0) 0x7fc6040b28a0 con 0x7fc6041430c0 2024-03-25T19:28:16.101 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.102+0000 7fc60895b640 10 monclient(hunting): _negotiate 2024-03-25T19:28:16.101 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.102+0000 7fc60895b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:16.102 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.102+0000 7fc60895b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:16.102 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.102+0000 7fc60895b640 10 monclient(hunting): my global_id is 4618 2024-03-25T19:28:16.102 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.102+0000 7fc60895b640 1 -- 172.21.15.83:0/3411840332 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc604142520 con 0x7fc6041430c0 2024-03-25T19:28:16.118 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.118+0000 7fc60895b640 1 -- 172.21.15.83:0/3411840332 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2145727448 0 0) 0x7fc604142520 con 0x7fc6041430c0 2024-03-25T19:28:16.118 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.118+0000 7fc60895b640 1 -- 172.21.15.83:0/3411840332 >> v1:172.21.15.83:6790/0 conn(0x7fc60413f710 legacy=0x7fc6041429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:16.118 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.118+0000 7fc60895b640 1 -- 172.21.15.83:0/3411840332 >> v1:172.21.15.83:6789/0 conn(0x7fc6040a98f0 legacy=0x7fc6040a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:16.119 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.118+0000 7fc60895b640 10 monclient: _finish_hunting 0 2024-03-25T19:28:16.119 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.118+0000 7fc60895b640 1 monclient: found mon.noname-c 2024-03-25T19:28:16.119 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.118+0000 7fc60895b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:16.119 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.118+0000 7fc60895b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.103:6789/0 2024-03-25T19:28:16.119 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.118+0000 7fc60895b640 1 -- 172.21.15.83:0/3411840332 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6040b44c0 con 0x7fc6041430c0 2024-03-25T19:28:16.119 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.118+0000 7fc60895b640 10 monclient: _finish_auth 0 2024-03-25T19:28:16.119 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.118+0000 7fc60895b640 10 monclient: _check_auth_tickets 2024-03-25T19:28:16.119 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.118+0000 7fc60895b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:16.119 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.118+0000 7fc60895b640 1 -- 172.21.15.83:0/3411840332 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fc5fc001930 con 0x7fc6041430c0 2024-03-25T19:28:16.119 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.118+0000 7fc60895b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:16.119 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.118+0000 7fc60895b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:28:16.119 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.118+0000 7fc60895b640 10 monclient: dump: 2024-03-25T19:28:16.119 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:28:16.119 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:28:16.119 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:16.120 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:16.120 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:16.120 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:28:16.120 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:28:16.120 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:28:16.120 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:28:16.120 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:16.120 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.118+0000 7fc60bbc1640 5 monclient: authenticate success, global_id 4618 2024-03-25T19:28:16.120 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.118+0000 7fc60bbc1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:16.140 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.141+0000 7fc60895b640 1 -- 172.21.15.83:0/3411840332 <== mon.1 v1:172.21.15.103:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5fc001a90 con 0x7fc6041430c0 2024-03-25T19:28:16.140 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.141+0000 7fc60895b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:16.140 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.141+0000 7fc60895b640 1 -- 172.21.15.83:0/3411840332 <== mon.1 v1:172.21.15.103:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fc5fc001ee0 con 0x7fc6041430c0 2024-03-25T19:28:16.140 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.141+0000 7fc60bbc1640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:16.141 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.141+0000 7fc60bbc1640 4 set_mon_vals no callback set 2024-03-25T19:28:16.141 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.141+0000 7fc60895b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:16.141 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.141+0000 7fc60895b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:16.141 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.141+0000 7fc60895b640 10 monclient: dump: 2024-03-25T19:28:16.141 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:28:16.141 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:28:16.141 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:16.141 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:16.141 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:16.141 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:28:16.141 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:28:16.141 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:28:16.141 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:28:16.141 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:16.143 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.141+0000 7fc60bbc1640 10 monclient: shutdown 2024-03-25T19:28:16.143 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.141+0000 7fc60bbc1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:16.143 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.141+0000 7fc60bbc1640 1 -- 172.21.15.83:0/3411840332 >> v1:172.21.15.103:6789/0 conn(0x7fc6041430c0 legacy=0x7fc60414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:16.143 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.142+0000 7fc60bbc1640 1 -- 172.21.15.83:0/3411840332 shutdown_connections 2024-03-25T19:28:16.143 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.142+0000 7fc60bbc1640 1 -- 172.21.15.83:0/3411840332 >> 172.21.15.83:0/3411840332 conn(0x7fc6040b7ea0 msgr2=0x7fc6040b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:16.143 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.142+0000 7fc60bbc1640 1 -- 172.21.15.83:0/3411840332 shutdown_connections 2024-03-25T19:28:16.143 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.142+0000 7fc60bbc1640 1 -- 172.21.15.83:0/3411840332 wait complete. 2024-03-25T19:28:16.143 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.143+0000 7fc60bbc1640 10 monclient: build_initial_monmap 2024-03-25T19:28:16.143 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.143+0000 7fc60bbc1640 10 monclient: monmap: 2024-03-25T19:28:16.143 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:28:16.144 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:16.144 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:28:16.144 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:28:16.144 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:16.144 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:28:16.144 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:28:16.144 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:28:16.144 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:28:16.144 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:16.144 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.143+0000 7fc60bbc1640 1 librados: starting msgr at 2024-03-25T19:28:16.144 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.143+0000 7fc60bbc1640 1 librados: starting objecter 2024-03-25T19:28:16.144 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.143+0000 7fc60bbc1640 1 Processor -- start 2024-03-25T19:28:16.144 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.143+0000 7fc60bbc1640 1 -- start start 2024-03-25T19:28:16.144 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.143+0000 7fc60bbc1640 1 librados: setting wanted keys 2024-03-25T19:28:16.144 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.143+0000 7fc60bbc1640 1 librados: calling monclient init 2024-03-25T19:28:16.144 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.143+0000 7fc60bbc1640 10 monclient: init 2024-03-25T19:28:16.145 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.144+0000 7fc60bbc1640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:16.146 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.144+0000 7fc60bbc1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:28:16.146 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.144+0000 7fc60bbc1640 10 monclient(hunting): picked mon.noname-a con 0x7fc6040a98f0 addr v1:172.21.15.83:6789/0 2024-03-25T19:28:16.146 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.144+0000 7fc60bbc1640 10 monclient(hunting): picked mon.noname-c con 0x7fc60413f710 addr v1:172.21.15.83:6790/0 2024-03-25T19:28:16.146 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.144+0000 7fc60bbc1640 10 monclient(hunting): picked mon.noname-b con 0x7fc6041430c0 addr v1:172.21.15.103:6789/0 2024-03-25T19:28:16.146 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.144+0000 7fc60bbc1640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6041d7800 con 0x7fc6040a98f0 2024-03-25T19:28:16.146 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.144+0000 7fc60bbc1640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6041d83a0 con 0x7fc6041430c0 2024-03-25T19:28:16.146 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.144+0000 7fc60bbc1640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6040b04d0 con 0x7fc60413f710 2024-03-25T19:28:16.146 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.144+0000 7fc60bbc1640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:16.146 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.144+0000 7fc60bbc1640 10 monclient(hunting): authenticate will time out at 6553.416072s 2024-03-25T19:28:16.146 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.147+0000 7fc60915c640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fc60413f710 0x7fc604138c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:33580/0 (socket says 172.21.15.83:33580) 2024-03-25T19:28:16.147 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.147+0000 7fc60915c640 1 -- 172.21.15.83:0/10284961 learned_addr learned my addr 172.21.15.83:0/10284961 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:28:16.148 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.148+0000 7fc5fa7fc640 10 client.?.objecter ms_handle_connect 0x7fc60413f710 2024-03-25T19:28:16.148 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.148+0000 7fc5fa7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:16.152 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.152+0000 7fc5fa7fc640 1 -- 172.21.15.83:0/10284961 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1274818603 0 0) 0x7fc6040b04d0 con 0x7fc60413f710 2024-03-25T19:28:16.152 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.152+0000 7fc5fa7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:28:16.152 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.152+0000 7fc5fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:16.152 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.152+0000 7fc5fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:16.152 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.152+0000 7fc5fa7fc640 10 monclient(hunting): my global_id is 4625 2024-03-25T19:28:16.152 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.152+0000 7fc5fa7fc640 1 -- 172.21.15.83:0/10284961 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5e8004eb0 con 0x7fc60413f710 2024-03-25T19:28:16.160 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.161+0000 7fc5fa7fc640 10 client.?.objecter ms_handle_connect 0x7fc6040a98f0 2024-03-25T19:28:16.161 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.161+0000 7fc5fa7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:16.161 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.161+0000 7fc5fa7fc640 1 -- 172.21.15.83:0/10284961 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2094144577 0 0) 0x7fc5e8004eb0 con 0x7fc60413f710 2024-03-25T19:28:16.161 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.161+0000 7fc5fa7fc640 1 -- 172.21.15.83:0/10284961 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6040b04d0 con 0x7fc60413f710 2024-03-25T19:28:16.161 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.161+0000 7fc5fa7fc640 1 -- 172.21.15.83:0/10284961 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fc5f40019b0 con 0x7fc60413f710 2024-03-25T19:28:16.161 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.161+0000 7fc5fa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:16.161 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.161+0000 7fc5fa7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:28:16.162 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.161+0000 7fc5fa7fc640 10 monclient(hunting): dump: 2024-03-25T19:28:16.162 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:28:16.162 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:28:16.162 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:16.162 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:16.162 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:16.162 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:28:16.162 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:28:16.162 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:28:16.162 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:28:16.162 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:16.162 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.161+0000 7fc5fa7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:16.166 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.167+0000 7fc5fa7fc640 10 client.?.objecter ms_handle_connect 0x7fc6041430c0 2024-03-25T19:28:16.166 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.167+0000 7fc5fa7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:16.173 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.173+0000 7fc5fa7fc640 1 -- 172.21.15.83:0/10284961 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3126903890 0 0) 0x7fc6040b04d0 con 0x7fc60413f710 2024-03-25T19:28:16.173 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.173+0000 7fc5fa7fc640 1 -- 172.21.15.83:0/10284961 >> v1:172.21.15.103:6789/0 conn(0x7fc6041430c0 legacy=0x7fc604139320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:16.173 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.173+0000 7fc5fa7fc640 1 -- 172.21.15.83:0/10284961 >> v1:172.21.15.83:6789/0 conn(0x7fc6040a98f0 legacy=0x7fc6041d7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:16.173 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.173+0000 7fc5fa7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:28:16.173 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.173+0000 7fc5fa7fc640 1 monclient: found mon.c 2024-03-25T19:28:16.173 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.173+0000 7fc5fa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:16.173 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.173+0000 7fc5fa7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:28:16.173 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.173+0000 7fc5fa7fc640 1 -- 172.21.15.83:0/10284961 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc604139a30 con 0x7fc60413f710 2024-03-25T19:28:16.174 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.173+0000 7fc5fa7fc640 10 monclient: _finish_auth 0 2024-03-25T19:28:16.174 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.173+0000 7fc5fa7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:28:16.174 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.173+0000 7fc5fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:16.174 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.173+0000 7fc60bbc1640 5 monclient: authenticate success, global_id 4625 2024-03-25T19:28:16.174 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.173+0000 7fc60bbc1640 10 monclient: _renew_subs 2024-03-25T19:28:16.174 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.173+0000 7fc60bbc1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:28:16.174 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.173+0000 7fc60bbc1640 1 -- 172.21.15.83:0/10284961 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6040b0740 con 0x7fc60413f710 2024-03-25T19:28:16.174 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.173+0000 7fc60bbc1640 10 client.4625.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:16.174 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.173+0000 7fc60bbc1640 10 monclient: _renew_subs 2024-03-25T19:28:16.174 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.173+0000 7fc60bbc1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:28:16.174 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.173+0000 7fc60bbc1640 1 -- 172.21.15.83:0/10284961 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc60414e070 con 0x7fc60413f710 2024-03-25T19:28:16.174 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.173+0000 7fc60bbc1640 1 librados: init done 2024-03-25T19:28:16.175 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.174+0000 7fc60bbc1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:16.175 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.174+0000 7fc60bbc1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:16.175 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.174+0000 7fc60bbc1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:28:16.175 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.174+0000 7fc60bbc1640 1 -- 172.21.15.83:0/10284961 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6040627b0 con 0x7fc60413f710 2024-03-25T19:28:16.475 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.475+0000 7fc5fa7fc640 1 -- 172.21.15.83:0/10284961 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5f4001c90 con 0x7fc60413f710 2024-03-25T19:28:16.475 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.475+0000 7fc5fa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:16.475 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.475+0000 7fc5fa7fc640 1 -- 172.21.15.83:0/10284961 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fc5f4001b10 con 0x7fc60413f710 2024-03-25T19:28:16.475 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.475+0000 7fc5fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:16.475 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.475+0000 7fc5fa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:28:16.475 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.475+0000 7fc5fa7fc640 10 monclient: dump: 2024-03-25T19:28:16.476 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:28:16.476 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:28:16.476 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:16.476 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:16.476 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:16.476 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:28:16.476 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:28:16.476 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:28:16.476 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:28:16.476 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:16.476 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.475+0000 7fc5fbfff640 4 set_mon_vals no callback set 2024-03-25T19:28:16.477 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.476+0000 7fc5fa7fc640 1 -- 172.21.15.83:0/10284961 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7fc5f4003b20 con 0x7fc60413f710 2024-03-25T19:28:16.477 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.477+0000 7fc60995d640 10 monclient: get_auth_request con 0x7fc5e806dd00 auth_method 0 2024-03-25T19:28:16.477 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.477+0000 7fc5fa7fc640 10 client.4625.objecter ms_handle_connect 0x7fc5e806dd00 2024-03-25T19:28:16.483 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.484+0000 7fc5fa7fc640 1 -- 172.21.15.83:0/10284961 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7fc5f4054850 con 0x7fc60413f710 2024-03-25T19:28:16.484 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.484+0000 7fc5fa7fc640 10 client.4625.objecter ms_dispatch 0x7fc604000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:28:16.484 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.484+0000 7fc5fa7fc640 3 client.4625.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:28:16.484 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.484+0000 7fc5fa7fc640 3 client.4625.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:28:16.484 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.484+0000 7fc5fa7fc640 20 client.4625.objecter dump_active .. 0 homeless 2024-03-25T19:28:16.484 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.484+0000 7fc5fa7fc640 1 -- 172.21.15.83:0/10284961 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc5f4050360 con 0x7fc60413f710 2024-03-25T19:28:16.484 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.484+0000 7fc5fa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:16.484 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.484+0000 7fc5fa7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:16.713 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.714+0000 7fc60bbc1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:16.714 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.714+0000 7fc60bbc1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:16.714 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.714+0000 7fc60bbc1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:28:16.714 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.714+0000 7fc60bbc1640 1 -- 172.21.15.83:0/10284961 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc6041d7490 con 0x7fc60413f710 2024-03-25T19:28:16.739 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.740+0000 7fc5fa7fc640 1 -- 172.21.15.83:0/10284961 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7fc5f4052290 con 0x7fc60413f710 2024-03-25T19:28:16.740 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.740+0000 7fc5fa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:16.740 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.740+0000 7fc5fa7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:28:16.740 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:28:16.740 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:28:16.743 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.742+0000 7fc60bbc1640 10 librados: watch_flush enter 2024-03-25T19:28:16.743 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.742+0000 7fc60bbc1640 10 librados: watch_flush exit 2024-03-25T19:28:16.743 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.742+0000 7fc60bbc1640 20 client.4625.objecter shutdown clearing up homeless session... 2024-03-25T19:28:16.743 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.742+0000 7fc60bbc1640 10 client.4625.objecter successfully canceled tick 2024-03-25T19:28:16.743 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.742+0000 7fc60bbc1640 1 -- 172.21.15.83:0/10284961 >> v1:172.21.15.103:6800/345072327 conn(0x7fc5e806dd00 legacy=0x7fc5e80701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:16.743 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.742+0000 7fc60bbc1640 10 monclient: shutdown 2024-03-25T19:28:16.743 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.742+0000 7fc60bbc1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:16.743 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.742+0000 7fc60bbc1640 1 -- 172.21.15.83:0/10284961 >> v1:172.21.15.83:6790/0 conn(0x7fc60413f710 legacy=0x7fc604138c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:16.743 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.742+0000 7fc60bbc1640 1 -- 172.21.15.83:0/10284961 shutdown_connections 2024-03-25T19:28:16.743 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.742+0000 7fc60bbc1640 1 -- 172.21.15.83:0/10284961 >> 172.21.15.83:0/10284961 conn(0x7fc6040b7ea0 msgr2=0x7fc604074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:16.743 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.743+0000 7fc60bbc1640 1 -- 172.21.15.83:0/10284961 shutdown_connections 2024-03-25T19:28:16.743 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.743+0000 7fc60bbc1640 1 -- 172.21.15.83:0/10284961 wait complete. 2024-03-25T19:28:16.743 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:16.743+0000 7fc60bbc1640 1 librados: shutdown 2024-03-25T19:28:16.755 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:28:23.562 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:28:23.676 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.675+0000 7fc1ae32b640 10 monclient: get_monmap_and_config 2024-03-25T19:28:23.676 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.675+0000 7fc1ae32b640 10 monclient: build_initial_monmap 2024-03-25T19:28:23.676 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.675+0000 7fc1ae32b640 10 monclient: monmap: 2024-03-25T19:28:23.676 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:28:23.676 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:28:23.676 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:28:23.676996+0000 2024-03-25T19:28:23.676 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:28:23.676996+0000 2024-03-25T19:28:23.676 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:23.676 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:28:23.676 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:28:23.676 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:28:23.677 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:28:23.677 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:23.678 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.675+0000 7fc1ae32b640 1 Processor -- start 2024-03-25T19:28:23.678 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.675+0000 7fc1ae32b640 1 -- start start 2024-03-25T19:28:23.678 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.675+0000 7fc1ae32b640 10 monclient: init 2024-03-25T19:28:23.678 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.676+0000 7fc1ae32b640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:23.678 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.676+0000 7fc1ae32b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:28:23.678 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.676+0000 7fc1ae32b640 10 monclient(hunting): picked mon.noname-c con 0x7fc1a80a98f0 addr v1:172.21.15.103:6789/0 2024-03-25T19:28:23.678 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.676+0000 7fc1ae32b640 10 monclient(hunting): picked mon.noname-b con 0x7fc1a813f710 addr v1:172.21.15.83:6790/0 2024-03-25T19:28:23.678 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.676+0000 7fc1ae32b640 10 monclient(hunting): picked mon.noname-a con 0x7fc1a81430c0 addr v1:172.21.15.83:6789/0 2024-03-25T19:28:23.678 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.676+0000 7fc1ae32b640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1a8142520 con 0x7fc1a81430c0 2024-03-25T19:28:23.678 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.676+0000 7fc1ae32b640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1a80b28a0 con 0x7fc1a80a98f0 2024-03-25T19:28:23.678 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.676+0000 7fc1ae32b640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1a80b2a30 con 0x7fc1a813f710 2024-03-25T19:28:23.679 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.676+0000 7fc1ae32b640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:23.679 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.676+0000 7fc1ae32b640 10 monclient(hunting): authenticate will time out at 6560.949290s 2024-03-25T19:28:23.686 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.685+0000 7fc1a77fe640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fc1a813f710 0x7fc1a81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:38436/0 (socket says 172.21.15.83:38436) 2024-03-25T19:28:23.686 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.685+0000 7fc1a77fe640 1 -- 172.21.15.83:0/1061513140 learned_addr learned my addr 172.21.15.83:0/1061513140 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:28:23.712 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.712+0000 7fc1a6ffd640 1 -- 172.21.15.83:0/1061513140 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2250331231 0 0) 0x7fc1a80b2a30 con 0x7fc1a813f710 2024-03-25T19:28:23.712 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.712+0000 7fc1a6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:23.713 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.712+0000 7fc1a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:23.713 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.712+0000 7fc1a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:23.713 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.712+0000 7fc1a6ffd640 10 monclient(hunting): my global_id is 4628 2024-03-25T19:28:23.713 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.712+0000 7fc1a6ffd640 1 -- 172.21.15.83:0/1061513140 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc190003520 con 0x7fc1a813f710 2024-03-25T19:28:23.713 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.712+0000 7fc1a6ffd640 1 -- 172.21.15.83:0/1061513140 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3130593129 0 0) 0x7fc1a80b28a0 con 0x7fc1a80a98f0 2024-03-25T19:28:23.713 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.712+0000 7fc1a6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:23.713 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.712+0000 7fc1a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:23.713 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.712+0000 7fc1a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:23.714 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.712+0000 7fc1a6ffd640 10 monclient(hunting): my global_id is 4624 2024-03-25T19:28:23.714 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.712+0000 7fc1a6ffd640 1 -- 172.21.15.83:0/1061513140 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1a80b2a30 con 0x7fc1a80a98f0 2024-03-25T19:28:23.714 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.713+0000 7fc1a6ffd640 1 -- 172.21.15.83:0/1061513140 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2654378120 0 0) 0x7fc1a8142520 con 0x7fc1a81430c0 2024-03-25T19:28:23.714 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.713+0000 7fc1a6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:23.714 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.713+0000 7fc1a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:23.714 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.713+0000 7fc1a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:23.714 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.713+0000 7fc1a6ffd640 10 monclient(hunting): my global_id is 4614 2024-03-25T19:28:23.714 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.713+0000 7fc1a6ffd640 1 -- 172.21.15.83:0/1061513140 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1a80b28a0 con 0x7fc1a81430c0 2024-03-25T19:28:23.724 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.723+0000 7fc1a6ffd640 1 -- 172.21.15.83:0/1061513140 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 500255434 0 0) 0x7fc1a80b28a0 con 0x7fc1a81430c0 2024-03-25T19:28:23.724 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.723+0000 7fc1a6ffd640 1 -- 172.21.15.83:0/1061513140 >> v1:172.21.15.83:6790/0 conn(0x7fc1a813f710 legacy=0x7fc1a81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:23.724 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.723+0000 7fc1a6ffd640 1 -- 172.21.15.83:0/1061513140 >> v1:172.21.15.103:6789/0 conn(0x7fc1a80a98f0 legacy=0x7fc1a80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:23.724 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.723+0000 7fc1a6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:28:23.724 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.723+0000 7fc1a6ffd640 1 monclient: found mon.noname-a 2024-03-25T19:28:23.724 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.723+0000 7fc1a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:23.724 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.723+0000 7fc1a6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.83:6789/0 2024-03-25T19:28:23.724 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.723+0000 7fc1a6ffd640 1 -- 172.21.15.83:0/1061513140 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1a80b44c0 con 0x7fc1a81430c0 2024-03-25T19:28:23.724 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.723+0000 7fc1a6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:28:23.724 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.723+0000 7fc1a6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:28:23.724 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.723+0000 7fc1a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:23.724 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.723+0000 7fc1a6ffd640 1 -- 172.21.15.83:0/1061513140 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fc1a0001930 con 0x7fc1a81430c0 2024-03-25T19:28:23.724 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.723+0000 7fc1ae32b640 5 monclient: authenticate success, global_id 4614 2024-03-25T19:28:23.724 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.723+0000 7fc1ae32b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:23.725 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.723+0000 7fc1a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:23.725 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.723+0000 7fc1a6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:28:23.725 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.723+0000 7fc1a6ffd640 10 monclient: dump: 2024-03-25T19:28:23.725 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:28:23.725 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:28:23.725 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:23.725 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:23.725 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:23.725 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:28:23.725 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:28:23.725 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:28:23.725 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:28:23.725 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:23.725 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.723+0000 7fc1ae32b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:23.761 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.761+0000 7fc1a6ffd640 1 -- 172.21.15.83:0/1061513140 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc1a0001a90 con 0x7fc1a81430c0 2024-03-25T19:28:23.761 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.761+0000 7fc1a6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:23.762 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.761+0000 7fc1a6ffd640 1 -- 172.21.15.83:0/1061513140 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fc1a0001ee0 con 0x7fc1a81430c0 2024-03-25T19:28:23.762 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.761+0000 7fc1ae32b640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:23.762 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.761+0000 7fc1ae32b640 4 set_mon_vals no callback set 2024-03-25T19:28:23.762 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.761+0000 7fc1a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:23.762 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.761+0000 7fc1a6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:23.762 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.761+0000 7fc1a6ffd640 10 monclient: dump: 2024-03-25T19:28:23.762 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:28:23.762 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:28:23.762 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:23.762 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:23.762 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:23.762 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:28:23.762 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:28:23.762 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:28:23.763 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:28:23.763 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:23.764 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.761+0000 7fc1ae32b640 10 monclient: shutdown 2024-03-25T19:28:23.764 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.761+0000 7fc1ae32b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:23.764 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.761+0000 7fc1ae32b640 1 -- 172.21.15.83:0/1061513140 >> v1:172.21.15.83:6789/0 conn(0x7fc1a81430c0 legacy=0x7fc1a814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:23.764 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.762+0000 7fc1ae32b640 1 -- 172.21.15.83:0/1061513140 shutdown_connections 2024-03-25T19:28:23.764 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.762+0000 7fc1ae32b640 1 -- 172.21.15.83:0/1061513140 >> 172.21.15.83:0/1061513140 conn(0x7fc1a80b7ea0 msgr2=0x7fc1a80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:23.764 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.762+0000 7fc1ae32b640 1 -- 172.21.15.83:0/1061513140 shutdown_connections 2024-03-25T19:28:23.764 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.762+0000 7fc1ae32b640 1 -- 172.21.15.83:0/1061513140 wait complete. 2024-03-25T19:28:23.765 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.762+0000 7fc1ae32b640 10 monclient: build_initial_monmap 2024-03-25T19:28:23.765 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.762+0000 7fc1ae32b640 10 monclient: monmap: 2024-03-25T19:28:23.765 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:28:23.765 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:23.765 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:28:23.765 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:28:23.765 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:23.765 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:28:23.765 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:28:23.765 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:28:23.765 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:28:23.765 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:23.765 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.763+0000 7fc1ae32b640 1 librados: starting msgr at 2024-03-25T19:28:23.766 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.763+0000 7fc1ae32b640 1 librados: starting objecter 2024-03-25T19:28:23.766 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.763+0000 7fc1ae32b640 1 Processor -- start 2024-03-25T19:28:23.766 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.763+0000 7fc1ae32b640 1 -- start start 2024-03-25T19:28:23.766 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.763+0000 7fc1ae32b640 1 librados: setting wanted keys 2024-03-25T19:28:23.766 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.763+0000 7fc1ae32b640 1 librados: calling monclient init 2024-03-25T19:28:23.766 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.763+0000 7fc1ae32b640 10 monclient: init 2024-03-25T19:28:23.767 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.764+0000 7fc1ae32b640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:23.767 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.764+0000 7fc1ae32b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:28:23.767 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.764+0000 7fc1ae32b640 10 monclient(hunting): picked mon.noname-b con 0x7fc1a80a98f0 addr v1:172.21.15.103:6789/0 2024-03-25T19:28:23.767 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.764+0000 7fc1ae32b640 10 monclient(hunting): picked mon.noname-c con 0x7fc1a813f710 addr v1:172.21.15.83:6790/0 2024-03-25T19:28:23.767 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.764+0000 7fc1ae32b640 10 monclient(hunting): picked mon.noname-a con 0x7fc1a81430c0 addr v1:172.21.15.83:6789/0 2024-03-25T19:28:23.767 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.764+0000 7fc1ae32b640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1a80b1710 con 0x7fc1a81430c0 2024-03-25T19:28:23.767 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.764+0000 7fc1ae32b640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1a81d83f0 con 0x7fc1a80a98f0 2024-03-25T19:28:23.767 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.764+0000 7fc1ae32b640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1a81d7ff0 con 0x7fc1a813f710 2024-03-25T19:28:23.767 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.764+0000 7fc1ae32b640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:23.767 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.764+0000 7fc1ae32b640 10 monclient(hunting): authenticate will time out at 6561.037210s 2024-03-25T19:28:23.782 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.782+0000 7fc1a7fff640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fc1a80a98f0 0x7fc1a81d78e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:34832/0 (socket says 172.21.15.83:34832) 2024-03-25T19:28:23.782 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.782+0000 7fc1a7fff640 1 -- 172.21.15.83:0/3501105910 learned_addr learned my addr 172.21.15.83:0/3501105910 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:28:23.784 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.783+0000 7fc1a4ff9640 10 client.?.objecter ms_handle_connect 0x7fc1a80a98f0 2024-03-25T19:28:23.784 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.783+0000 7fc1a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:23.788 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.787+0000 7fc1a4ff9640 10 client.?.objecter ms_handle_connect 0x7fc1a81430c0 2024-03-25T19:28:23.788 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.787+0000 7fc1a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:23.792 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.792+0000 7fc1a4ff9640 10 client.?.objecter ms_handle_connect 0x7fc1a813f710 2024-03-25T19:28:23.792 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.792+0000 7fc1a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:23.799 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.799+0000 7fc1a4ff9640 1 -- 172.21.15.83:0/3501105910 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2932034171 0 0) 0x7fc1a80b1710 con 0x7fc1a81430c0 2024-03-25T19:28:23.800 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.799+0000 7fc1a4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:28:23.800 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.799+0000 7fc1a4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:23.800 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.799+0000 7fc1a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:23.800 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.799+0000 7fc1a4ff9640 10 monclient(hunting): my global_id is 4617 2024-03-25T19:28:23.800 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.799+0000 7fc1a4ff9640 1 -- 172.21.15.83:0/3501105910 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc194004eb0 con 0x7fc1a81430c0 2024-03-25T19:28:23.807 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.807+0000 7fc1a4ff9640 1 -- 172.21.15.83:0/3501105910 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2645636890 0 0) 0x7fc1a81d7ff0 con 0x7fc1a813f710 2024-03-25T19:28:23.807 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.807+0000 7fc1a4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:28:23.807 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.807+0000 7fc1a4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:23.808 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.807+0000 7fc1a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:23.808 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.807+0000 7fc1a4ff9640 10 monclient(hunting): my global_id is 4631 2024-03-25T19:28:23.808 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.807+0000 7fc1a4ff9640 1 -- 172.21.15.83:0/3501105910 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1a80b1710 con 0x7fc1a813f710 2024-03-25T19:28:23.814 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.813+0000 7fc1a4ff9640 1 -- 172.21.15.83:0/3501105910 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3526682696 0 0) 0x7fc1a81d83f0 con 0x7fc1a80a98f0 2024-03-25T19:28:23.814 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.813+0000 7fc1a4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:28:23.814 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.813+0000 7fc1a4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:23.814 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.813+0000 7fc1a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:23.814 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.813+0000 7fc1a4ff9640 10 monclient(hunting): my global_id is 4627 2024-03-25T19:28:23.814 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.813+0000 7fc1a4ff9640 1 -- 172.21.15.83:0/3501105910 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1a81d7ff0 con 0x7fc1a80a98f0 2024-03-25T19:28:23.826 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.825+0000 7fc1a4ff9640 1 -- 172.21.15.83:0/3501105910 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2920633630 0 0) 0x7fc1a80b1710 con 0x7fc1a813f710 2024-03-25T19:28:23.826 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.825+0000 7fc1a4ff9640 1 -- 172.21.15.83:0/3501105910 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc1a81d83f0 con 0x7fc1a813f710 2024-03-25T19:28:23.826 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.825+0000 7fc1a4ff9640 1 -- 172.21.15.83:0/3501105910 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fc19c001950 con 0x7fc1a813f710 2024-03-25T19:28:23.826 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.825+0000 7fc1a4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:23.827 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.825+0000 7fc1a4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:28:23.827 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.825+0000 7fc1a4ff9640 10 monclient(hunting): dump: 2024-03-25T19:28:23.827 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:28:23.827 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:28:23.827 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:23.827 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:23.827 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:23.827 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:28:23.827 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:28:23.827 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:28:23.827 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:28:23.827 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:23.827 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.825+0000 7fc1a4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:23.828 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.826+0000 7fc1a4ff9640 1 -- 172.21.15.83:0/3501105910 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2685459162 0 0) 0x7fc1a81d7ff0 con 0x7fc1a80a98f0 2024-03-25T19:28:23.828 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.826+0000 7fc1a4ff9640 1 -- 172.21.15.83:0/3501105910 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc1a80b1710 con 0x7fc1a80a98f0 2024-03-25T19:28:23.828 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.826+0000 7fc1a4ff9640 1 -- 172.21.15.83:0/3501105910 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fc1980019b0 con 0x7fc1a80a98f0 2024-03-25T19:28:23.828 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.826+0000 7fc1a4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:23.828 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.826+0000 7fc1a4ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:23.828 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.826+0000 7fc1a4ff9640 10 monclient(hunting): dump: 2024-03-25T19:28:23.828 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:28:23.829 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:28:23.829 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:23.829 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:23.829 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:23.829 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:28:23.829 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:28:23.829 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:28:23.829 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:28:23.829 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:23.835 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.835+0000 7fc1a7fff640 1 -- 172.21.15.83:0/3501105910 >> v1:172.21.15.103:6789/0 conn(0x7fc1a80a98f0 legacy=0x7fc1a81d78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-25T19:28:23.835 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.835+0000 7fc1a7fff640 1 -- 172.21.15.83:0/3501105910 >> v1:172.21.15.103:6789/0 conn(0x7fc1a80a98f0 legacy=0x7fc1a81d78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:28:23.836 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.835+0000 7fc1a7fff640 1 --1- 172.21.15.83:0/3501105910 >> v1:172.21.15.103:6789/0 conn(0x7fc1a80a98f0 0x7fc1a81d78e0 :-1 s=OPENED pgs=63224 cs=1 l=1).handle_message read tag failed 2024-03-25T19:28:23.836 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.835+0000 7fc1a7fff640 1 --1- 172.21.15.83:0/3501105910 >> v1:172.21.15.103:6789/0 conn(0x7fc1a80a98f0 0x7fc1a81d78e0 :-1 s=OPENED pgs=63224 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:28:23.836 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.835+0000 7fc1a4ff9640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.103:6789/0 2024-03-25T19:28:23.846 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.844+0000 7fc1a4ff9640 1 -- 172.21.15.83:0/3501105910 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3885612666 0 0) 0x7fc194004eb0 con 0x7fc1a81430c0 2024-03-25T19:28:23.846 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.844+0000 7fc1a4ff9640 1 -- 172.21.15.83:0/3501105910 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc1a81d7ff0 con 0x7fc1a81430c0 2024-03-25T19:28:23.846 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.844+0000 7fc1a4ff9640 1 -- 172.21.15.83:0/3501105910 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fc1a0001de0 con 0x7fc1a81430c0 2024-03-25T19:28:23.846 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.844+0000 7fc1a4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:23.846 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.844+0000 7fc1a4ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:23.846 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.844+0000 7fc1a4ff9640 10 monclient(hunting): dump: 2024-03-25T19:28:23.846 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:28:23.846 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:28:23.846 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:23.846 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:23.846 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:23.846 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:28:23.846 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:28:23.847 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:28:23.847 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:28:23.847 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:23.853 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.852+0000 7fc1a4ff9640 1 -- 172.21.15.83:0/3501105910 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 889992945 0 0) 0x7fc1a81d7ff0 con 0x7fc1a81430c0 2024-03-25T19:28:23.853 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.852+0000 7fc1a4ff9640 1 -- 172.21.15.83:0/3501105910 >> v1:172.21.15.83:6790/0 conn(0x7fc1a813f710 legacy=0x7fc1a814d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:23.854 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.852+0000 7fc1a4ff9640 1 -- 172.21.15.83:0/3501105910 >> v1:172.21.15.103:6789/0 conn(0x7fc1a80a98f0 legacy=0x7fc1a81d78e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:28:23.854 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.852+0000 7fc1a4ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:28:23.854 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.852+0000 7fc1a4ff9640 1 monclient: found mon.a 2024-03-25T19:28:23.854 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.852+0000 7fc1a4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:23.854 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.852+0000 7fc1a4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:28:23.854 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.852+0000 7fc1a4ff9640 1 -- 172.21.15.83:0/3501105910 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1a81d8180 con 0x7fc1a81430c0 2024-03-25T19:28:23.854 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.852+0000 7fc1a4ff9640 10 monclient: _finish_auth 0 2024-03-25T19:28:23.854 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.852+0000 7fc1a4ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:28:23.854 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.852+0000 7fc1a4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:23.854 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.852+0000 7fc1ae32b640 5 monclient: authenticate success, global_id 4617 2024-03-25T19:28:23.854 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.852+0000 7fc1ae32b640 10 monclient: _renew_subs 2024-03-25T19:28:23.854 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.852+0000 7fc1ae32b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:28:23.854 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.852+0000 7fc1ae32b640 1 -- 172.21.15.83:0/3501105910 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1a814e0d0 con 0x7fc1a81430c0 2024-03-25T19:28:23.855 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.852+0000 7fc1ae32b640 10 client.4617.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:23.855 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.852+0000 7fc1ae32b640 10 monclient: _renew_subs 2024-03-25T19:28:23.855 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.852+0000 7fc1ae32b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:28:23.855 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.852+0000 7fc1ae32b640 1 -- 172.21.15.83:0/3501105910 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1a814e3d0 con 0x7fc1a81430c0 2024-03-25T19:28:23.855 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.852+0000 7fc1ae32b640 1 librados: init done 2024-03-25T19:28:23.855 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.853+0000 7fc1ae32b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:23.856 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.853+0000 7fc1ae32b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:23.856 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.853+0000 7fc1ae32b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:28:23.856 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:23.853+0000 7fc1ae32b640 1 -- 172.21.15.83:0/3501105910 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1a814ed60 con 0x7fc1a81430c0 2024-03-25T19:28:24.154 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:24.153+0000 7fc1a4ff9640 1 -- 172.21.15.83:0/3501105910 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc1a0002020 con 0x7fc1a81430c0 2024-03-25T19:28:24.155 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:24.153+0000 7fc1a4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:24.155 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:24.153+0000 7fc1a4ff9640 1 -- 172.21.15.83:0/3501105910 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fc1a0003dc0 con 0x7fc1a81430c0 2024-03-25T19:28:24.155 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:24.154+0000 7fc1a4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:24.155 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:24.154+0000 7fc1a4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:24.155 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:24.154+0000 7fc1a4ff9640 10 monclient: dump: 2024-03-25T19:28:24.155 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:28:24.155 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:28:24.155 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:24.155 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:24.155 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:24.156 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:28:24.156 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:28:24.156 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:28:24.156 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:28:24.156 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:24.156 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:24.153+0000 7fc1a67fc640 4 set_mon_vals no callback set 2024-03-25T19:28:24.157 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:24.154+0000 7fc1a4ff9640 1 -- 172.21.15.83:0/3501105910 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7fc1a0019d90 con 0x7fc1a81430c0 2024-03-25T19:28:24.157 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:24.155+0000 7fc1a7fff640 10 monclient: get_auth_request con 0x7fc19406e160 auth_method 0 2024-03-25T19:28:24.157 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:24.155+0000 7fc1a4ff9640 10 client.4617.objecter ms_handle_connect 0x7fc19406e160 2024-03-25T19:28:24.164 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:24.163+0000 7fc1a4ff9640 1 -- 172.21.15.83:0/3501105910 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7fc1a0053870 con 0x7fc1a81430c0 2024-03-25T19:28:24.164 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:24.163+0000 7fc1a4ff9640 10 client.4617.objecter ms_dispatch 0x7fc1a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:28:24.164 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:24.163+0000 7fc1a4ff9640 3 client.4617.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:28:24.164 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:24.163+0000 7fc1a4ff9640 3 client.4617.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:28:24.164 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:24.163+0000 7fc1a4ff9640 20 client.4617.objecter dump_active .. 0 homeless 2024-03-25T19:28:24.164 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:24.163+0000 7fc1a4ff9640 1 -- 172.21.15.83:0/3501105910 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc1a004f380 con 0x7fc1a81430c0 2024-03-25T19:28:24.164 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:24.163+0000 7fc1a4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:24.164 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:24.163+0000 7fc1a4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:24.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:24.385+0000 7fc1ae32b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:24.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:24.385+0000 7fc1ae32b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:24.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:24.385+0000 7fc1ae32b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:28:24.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:24.385+0000 7fc1ae32b640 1 -- 172.21.15.83:0/3501105910 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc1a81d8580 con 0x7fc1a81430c0 2024-03-25T19:28:24.423 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:24.423+0000 7fc1a4ff9640 1 -- 172.21.15.83:0/3501105910 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7fc1a00512b0 con 0x7fc1a81430c0 2024-03-25T19:28:24.423 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:24.423+0000 7fc1a4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:24.423 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:24.423+0000 7fc1a4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:28:24.423 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:28:24.423 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:28:24.425 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:24.425+0000 7fc1ae32b640 10 librados: watch_flush enter 2024-03-25T19:28:24.426 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:24.425+0000 7fc1ae32b640 10 librados: watch_flush exit 2024-03-25T19:28:24.426 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:24.425+0000 7fc1ae32b640 20 client.4617.objecter shutdown clearing up homeless session... 2024-03-25T19:28:24.426 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:24.425+0000 7fc1ae32b640 10 client.4617.objecter successfully canceled tick 2024-03-25T19:28:24.426 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:24.425+0000 7fc1ae32b640 1 -- 172.21.15.83:0/3501105910 >> v1:172.21.15.103:6800/345072327 conn(0x7fc19406e160 legacy=0x7fc194070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:24.426 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:24.425+0000 7fc1ae32b640 10 monclient: shutdown 2024-03-25T19:28:24.426 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:24.425+0000 7fc1ae32b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:24.426 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:24.425+0000 7fc1ae32b640 1 -- 172.21.15.83:0/3501105910 >> v1:172.21.15.83:6789/0 conn(0x7fc1a81430c0 legacy=0x7fc1a81d4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:24.426 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:24.426+0000 7fc1ae32b640 1 -- 172.21.15.83:0/3501105910 shutdown_connections 2024-03-25T19:28:24.426 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:24.426+0000 7fc1ae32b640 1 -- 172.21.15.83:0/3501105910 >> 172.21.15.83:0/3501105910 conn(0x7fc1a80b7ea0 msgr2=0x7fc1a8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:24.426 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:24.426+0000 7fc1ae32b640 1 -- 172.21.15.83:0/3501105910 shutdown_connections 2024-03-25T19:28:24.427 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:24.426+0000 7fc1ae32b640 1 -- 172.21.15.83:0/3501105910 wait complete. 2024-03-25T19:28:24.427 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:24.426+0000 7fc1ae32b640 1 librados: shutdown 2024-03-25T19:28:24.438 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:28:28.776 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:09:10.703 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:28.784 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:09:10.712 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:28.806 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:09:10.733 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:28.833 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:09:10.761 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:28.835 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:09:10.745 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:28.839 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:09:10.750 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:28.844 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:09:10.771 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:28.875 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:09:10.785 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:28.881 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:09:10.809 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:31.245 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:28:31.359 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.359+0000 7f0f43406640 10 monclient: get_monmap_and_config 2024-03-25T19:28:31.360 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.359+0000 7f0f43406640 10 monclient: build_initial_monmap 2024-03-25T19:28:31.360 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.360+0000 7f0f43406640 10 monclient: monmap: 2024-03-25T19:28:31.360 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:28:31.360 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:28:31.360 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:28:31.360885+0000 2024-03-25T19:28:31.360 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:28:31.360885+0000 2024-03-25T19:28:31.360 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:31.360 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:28:31.360 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:28:31.360 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:28:31.360 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:28:31.360 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:31.362 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.360+0000 7f0f43406640 1 Processor -- start 2024-03-25T19:28:31.362 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.360+0000 7f0f43406640 1 -- start start 2024-03-25T19:28:31.362 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.360+0000 7f0f43406640 10 monclient: init 2024-03-25T19:28:31.362 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.361+0000 7f0f43406640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:31.362 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.361+0000 7f0f43406640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:28:31.362 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.361+0000 7f0f43406640 10 monclient(hunting): picked mon.noname-a con 0x7f0f3c0704a0 addr v1:172.21.15.83:6789/0 2024-03-25T19:28:31.362 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.361+0000 7f0f43406640 10 monclient(hunting): picked mon.noname-b con 0x7f0f3c070880 addr v1:172.21.15.83:6790/0 2024-03-25T19:28:31.362 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.361+0000 7f0f43406640 10 monclient(hunting): picked mon.noname-c con 0x7f0f3c149a40 addr v1:172.21.15.103:6789/0 2024-03-25T19:28:31.362 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.361+0000 7f0f43406640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f3c143870 con 0x7f0f3c0704a0 2024-03-25T19:28:31.362 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.361+0000 7f0f43406640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f3c072480 con 0x7f0f3c149a40 2024-03-25T19:28:31.362 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.361+0000 7f0f43406640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f3c072760 con 0x7f0f3c070880 2024-03-25T19:28:31.363 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.361+0000 7f0f43406640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:31.363 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.361+0000 7f0f43406640 10 monclient(hunting): authenticate will time out at 6568.633339s 2024-03-25T19:28:31.365 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.366+0000 7f0f411a2640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f0f3c0704a0 0x7f0f3c143d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:59238/0 (socket says 172.21.15.83:59238) 2024-03-25T19:28:31.366 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.366+0000 7f0f411a2640 1 -- 172.21.15.83:0/1255115971 learned_addr learned my addr 172.21.15.83:0/1255115971 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:28:31.376 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.376+0000 7f0f33fff640 1 -- 172.21.15.83:0/1255115971 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1981448828 0 0) 0x7f0f3c072480 con 0x7f0f3c149a40 2024-03-25T19:28:31.376 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.376+0000 7f0f33fff640 10 monclient(hunting): _negotiate 2024-03-25T19:28:31.376 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.376+0000 7f0f33fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:31.376 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.376+0000 7f0f33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:31.377 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.376+0000 7f0f33fff640 10 monclient(hunting): my global_id is 4630 2024-03-25T19:28:31.377 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.376+0000 7f0f33fff640 1 -- 172.21.15.83:0/1255115971 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f28003520 con 0x7f0f3c149a40 2024-03-25T19:28:31.377 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.377+0000 7f0f33fff640 1 -- 172.21.15.83:0/1255115971 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 782942779 0 0) 0x7f0f3c143870 con 0x7f0f3c0704a0 2024-03-25T19:28:31.377 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.377+0000 7f0f33fff640 10 monclient(hunting): _negotiate 2024-03-25T19:28:31.377 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.377+0000 7f0f33fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:31.377 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.377+0000 7f0f33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:31.377 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.377+0000 7f0f33fff640 10 monclient(hunting): my global_id is 4620 2024-03-25T19:28:31.378 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.377+0000 7f0f33fff640 1 -- 172.21.15.83:0/1255115971 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f3c072480 con 0x7f0f3c0704a0 2024-03-25T19:28:31.384 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.384+0000 7f0f33fff640 1 -- 172.21.15.83:0/1255115971 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1983550922 0 0) 0x7f0f3c072480 con 0x7f0f3c0704a0 2024-03-25T19:28:31.384 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.384+0000 7f0f33fff640 1 -- 172.21.15.83:0/1255115971 >> v1:172.21.15.83:6790/0 conn(0x7f0f3c070880 legacy=0x7f0f3c13d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:31.384 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.384+0000 7f0f33fff640 1 -- 172.21.15.83:0/1255115971 >> v1:172.21.15.103:6789/0 conn(0x7f0f3c149a40 legacy=0x7f0f3c14be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:31.384 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.384+0000 7f0f33fff640 10 monclient: _finish_hunting 0 2024-03-25T19:28:31.384 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.384+0000 7f0f33fff640 1 monclient: found mon.noname-a 2024-03-25T19:28:31.384 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.384+0000 7f0f33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:31.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.384+0000 7f0f33fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.83:6789/0 2024-03-25T19:28:31.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.384+0000 7f0f33fff640 1 -- 172.21.15.83:0/1255115971 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f3c072a40 con 0x7f0f3c0704a0 2024-03-25T19:28:31.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.384+0000 7f0f33fff640 10 monclient: _finish_auth 0 2024-03-25T19:28:31.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.384+0000 7f0f33fff640 10 monclient: _check_auth_tickets 2024-03-25T19:28:31.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.384+0000 7f0f33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:31.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.384+0000 7f0f33fff640 1 -- 172.21.15.83:0/1255115971 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f0f38001670 con 0x7f0f3c0704a0 2024-03-25T19:28:31.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.384+0000 7f0f33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:31.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.384+0000 7f0f33fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:28:31.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.384+0000 7f0f33fff640 10 monclient: dump: 2024-03-25T19:28:31.385 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:28:31.385 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:28:31.385 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:31.385 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:31.386 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:31.386 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:28:31.386 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:28:31.386 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:28:31.386 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:28:31.386 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:31.386 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.384+0000 7f0f43406640 5 monclient: authenticate success, global_id 4620 2024-03-25T19:28:31.386 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.384+0000 7f0f43406640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:31.393 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.393+0000 7f0f33fff640 1 -- 172.21.15.83:0/1255115971 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f380017d0 con 0x7f0f3c0704a0 2024-03-25T19:28:31.393 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.393+0000 7f0f33fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:31.393 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.393+0000 7f0f33fff640 1 -- 172.21.15.83:0/1255115971 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f0f38001ba0 con 0x7f0f3c0704a0 2024-03-25T19:28:31.393 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.393+0000 7f0f33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:31.393 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.393+0000 7f0f33fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:31.393 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.393+0000 7f0f33fff640 10 monclient: dump: 2024-03-25T19:28:31.393 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:28:31.394 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:28:31.394 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:31.394 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:31.394 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:31.394 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:28:31.394 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:28:31.394 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:28:31.394 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:28:31.394 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:31.394 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.393+0000 7f0f43406640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:31.394 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.393+0000 7f0f43406640 4 set_mon_vals no callback set 2024-03-25T19:28:31.396 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.394+0000 7f0f43406640 10 monclient: shutdown 2024-03-25T19:28:31.396 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.394+0000 7f0f43406640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:31.396 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.394+0000 7f0f43406640 1 -- 172.21.15.83:0/1255115971 >> v1:172.21.15.83:6789/0 conn(0x7f0f3c0704a0 legacy=0x7f0f3c143d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:31.396 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.394+0000 7f0f43406640 1 -- 172.21.15.83:0/1255115971 shutdown_connections 2024-03-25T19:28:31.396 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.394+0000 7f0f43406640 1 -- 172.21.15.83:0/1255115971 >> 172.21.15.83:0/1255115971 conn(0x7f0f3c0b5c00 msgr2=0x7f0f3c0b5fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:31.396 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.394+0000 7f0f43406640 1 -- 172.21.15.83:0/1255115971 shutdown_connections 2024-03-25T19:28:31.396 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.394+0000 7f0f43406640 1 -- 172.21.15.83:0/1255115971 wait complete. 2024-03-25T19:28:31.397 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.395+0000 7f0f43406640 10 monclient: build_initial_monmap 2024-03-25T19:28:31.397 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.395+0000 7f0f43406640 10 monclient: monmap: 2024-03-25T19:28:31.397 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:28:31.397 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:31.397 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:28:31.397 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:28:31.397 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:31.397 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:28:31.397 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:28:31.397 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:28:31.397 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:28:31.397 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:31.397 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.396+0000 7f0f43406640 1 librados: starting msgr at 2024-03-25T19:28:31.397 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.396+0000 7f0f43406640 1 librados: starting objecter 2024-03-25T19:28:31.398 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.396+0000 7f0f43406640 1 Processor -- start 2024-03-25T19:28:31.398 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.396+0000 7f0f43406640 1 -- start start 2024-03-25T19:28:31.398 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.396+0000 7f0f43406640 1 librados: setting wanted keys 2024-03-25T19:28:31.398 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.396+0000 7f0f43406640 1 librados: calling monclient init 2024-03-25T19:28:31.398 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.396+0000 7f0f43406640 10 monclient: init 2024-03-25T19:28:31.399 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.397+0000 7f0f43406640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:31.399 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.397+0000 7f0f43406640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:28:31.399 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.397+0000 7f0f43406640 10 monclient(hunting): picked mon.noname-b con 0x7f0f3c0704a0 addr v1:172.21.15.103:6789/0 2024-03-25T19:28:31.399 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.397+0000 7f0f43406640 10 monclient(hunting): picked mon.noname-a con 0x7f0f3c070880 addr v1:172.21.15.83:6789/0 2024-03-25T19:28:31.399 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.397+0000 7f0f43406640 10 monclient(hunting): picked mon.noname-c con 0x7f0f3c149a40 addr v1:172.21.15.83:6790/0 2024-03-25T19:28:31.399 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.397+0000 7f0f43406640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f3c138b50 con 0x7f0f3c070880 2024-03-25T19:28:31.399 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.397+0000 7f0f43406640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f3c139e90 con 0x7f0f3c0704a0 2024-03-25T19:28:31.399 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.397+0000 7f0f43406640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f3c13a100 con 0x7f0f3c149a40 2024-03-25T19:28:31.399 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.397+0000 7f0f43406640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:31.399 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.397+0000 7f0f43406640 10 monclient(hunting): authenticate will time out at 6568.669070s 2024-03-25T19:28:31.402 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.403+0000 7f0f419a3640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f0f3c149a40 0x7f0f3c1dfaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:35912/0 (socket says 172.21.15.83:35912) 2024-03-25T19:28:31.402 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.403+0000 7f0f419a3640 1 -- 172.21.15.83:0/1844987133 learned_addr learned my addr 172.21.15.83:0/1844987133 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:28:31.404 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.405+0000 7f0f31ffb640 10 client.?.objecter ms_handle_connect 0x7f0f3c149a40 2024-03-25T19:28:31.404 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.405+0000 7f0f31ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:31.414 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.415+0000 7f0f31ffb640 10 client.?.objecter ms_handle_connect 0x7f0f3c070880 2024-03-25T19:28:31.414 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.415+0000 7f0f31ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:31.416 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.416+0000 7f0f31ffb640 10 client.?.objecter ms_handle_connect 0x7f0f3c0704a0 2024-03-25T19:28:31.416 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.416+0000 7f0f31ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:31.418 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.418+0000 7f0f31ffb640 1 -- 172.21.15.83:0/1844987133 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3287239054 0 0) 0x7f0f3c13a100 con 0x7f0f3c149a40 2024-03-25T19:28:31.418 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.418+0000 7f0f31ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:28:31.418 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.418+0000 7f0f31ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:31.418 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.418+0000 7f0f31ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:31.418 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.418+0000 7f0f31ffb640 10 monclient(hunting): my global_id is 4637 2024-03-25T19:28:31.418 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.418+0000 7f0f31ffb640 1 -- 172.21.15.83:0/1844987133 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f20004eb0 con 0x7f0f3c149a40 2024-03-25T19:28:31.431 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.431+0000 7f0f31ffb640 1 -- 172.21.15.83:0/1844987133 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3540969537 0 0) 0x7f0f3c139e90 con 0x7f0f3c0704a0 2024-03-25T19:28:31.431 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.432+0000 7f0f31ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:28:31.431 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.432+0000 7f0f31ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:31.432 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.432+0000 7f0f31ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:31.432 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.432+0000 7f0f31ffb640 10 monclient(hunting): my global_id is 4633 2024-03-25T19:28:31.432 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.432+0000 7f0f31ffb640 1 -- 172.21.15.83:0/1844987133 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f3c13a100 con 0x7f0f3c0704a0 2024-03-25T19:28:31.435 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.436+0000 7f0f31ffb640 1 -- 172.21.15.83:0/1844987133 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 483991811 0 0) 0x7f0f3c138b50 con 0x7f0f3c070880 2024-03-25T19:28:31.435 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.436+0000 7f0f31ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:28:31.435 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.436+0000 7f0f31ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:31.436 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.436+0000 7f0f31ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:31.436 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.436+0000 7f0f31ffb640 10 monclient(hunting): my global_id is 4623 2024-03-25T19:28:31.436 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.436+0000 7f0f31ffb640 1 -- 172.21.15.83:0/1844987133 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f3c139e90 con 0x7f0f3c070880 2024-03-25T19:28:31.442 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.443+0000 7f0f31ffb640 1 -- 172.21.15.83:0/1844987133 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 475322729 0 0) 0x7f0f3c13a100 con 0x7f0f3c0704a0 2024-03-25T19:28:31.443 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.443+0000 7f0f31ffb640 1 -- 172.21.15.83:0/1844987133 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f3c138b50 con 0x7f0f3c0704a0 2024-03-25T19:28:31.443 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.443+0000 7f0f31ffb640 1 -- 172.21.15.83:0/1844987133 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f0f38001ae0 con 0x7f0f3c0704a0 2024-03-25T19:28:31.443 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.443+0000 7f0f31ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:31.443 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.443+0000 7f0f31ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:28:31.443 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.443+0000 7f0f31ffb640 10 monclient(hunting): dump: 2024-03-25T19:28:31.443 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:28:31.443 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:28:31.443 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:31.443 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:31.443 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:31.443 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:28:31.443 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:28:31.443 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:28:31.443 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:28:31.443 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:31.444 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.443+0000 7f0f31ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:31.446 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.446+0000 7f0f31ffb640 1 -- 172.21.15.83:0/1844987133 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4139523016 0 0) 0x7f0f20004eb0 con 0x7f0f3c149a40 2024-03-25T19:28:31.446 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.446+0000 7f0f31ffb640 1 -- 172.21.15.83:0/1844987133 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f3c13a100 con 0x7f0f3c149a40 2024-03-25T19:28:31.446 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.446+0000 7f0f31ffb640 1 -- 172.21.15.83:0/1844987133 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f0f34001c30 con 0x7f0f3c149a40 2024-03-25T19:28:31.446 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.446+0000 7f0f31ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:31.446 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.446+0000 7f0f31ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:28:31.446 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.446+0000 7f0f31ffb640 10 monclient(hunting): dump: 2024-03-25T19:28:31.446 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:28:31.446 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:28:31.446 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:31.446 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:31.446 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:31.446 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:28:31.447 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:28:31.447 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:28:31.447 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:28:31.447 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:31.457 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.456+0000 7f0f31ffb640 1 -- 172.21.15.83:0/1844987133 <== mon.1 v1:172.21.15.103:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 340519261 0 0) 0x7f0f3c138b50 con 0x7f0f3c0704a0 2024-03-25T19:28:31.457 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.457+0000 7f0f31ffb640 1 -- 172.21.15.83:0/1844987133 >> v1:172.21.15.83:6790/0 conn(0x7f0f3c149a40 legacy=0x7f0f3c1dfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:31.457 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.457+0000 7f0f31ffb640 1 -- 172.21.15.83:0/1844987133 >> v1:172.21.15.83:6789/0 conn(0x7f0f3c070880 legacy=0x7f0f3c1d84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:31.457 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.457+0000 7f0f31ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:28:31.457 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.457+0000 7f0f31ffb640 1 monclient: found mon.b 2024-03-25T19:28:31.457 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.457+0000 7f0f31ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:31.457 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.457+0000 7f0f31ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:28:31.457 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.457+0000 7f0f31ffb640 1 -- 172.21.15.83:0/1844987133 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f3c13a5b0 con 0x7f0f3c0704a0 2024-03-25T19:28:31.457 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.457+0000 7f0f31ffb640 10 monclient: _finish_auth 0 2024-03-25T19:28:31.457 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.457+0000 7f0f31ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:28:31.457 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.457+0000 7f0f31ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:31.458 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.457+0000 7f0f43406640 5 monclient: authenticate success, global_id 4633 2024-03-25T19:28:31.458 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.457+0000 7f0f43406640 10 monclient: _renew_subs 2024-03-25T19:28:31.458 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.457+0000 7f0f43406640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:28:31.458 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.457+0000 7f0f43406640 1 -- 172.21.15.83:0/1844987133 --> v1:172.21.15.103:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f3c13a860 con 0x7f0f3c0704a0 2024-03-25T19:28:31.458 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.457+0000 7f0f43406640 10 client.4633.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:31.458 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.457+0000 7f0f43406640 10 monclient: _renew_subs 2024-03-25T19:28:31.458 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.457+0000 7f0f43406640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:28:31.458 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.457+0000 7f0f43406640 1 -- 172.21.15.83:0/1844987133 --> v1:172.21.15.103:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f3c1d8dd0 con 0x7f0f3c0704a0 2024-03-25T19:28:31.458 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.457+0000 7f0f43406640 1 librados: init done 2024-03-25T19:28:31.459 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.458+0000 7f0f43406640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:31.459 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.458+0000 7f0f43406640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:31.459 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.458+0000 7f0f43406640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:28:31.459 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.458+0000 7f0f43406640 1 -- 172.21.15.83:0/1844987133 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f3c1d9080 con 0x7f0f3c0704a0 2024-03-25T19:28:31.736 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.735+0000 7f0f31ffb640 1 -- 172.21.15.83:0/1844987133 <== mon.1 v1:172.21.15.103:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f38001c40 con 0x7f0f3c0704a0 2024-03-25T19:28:31.736 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.735+0000 7f0f31ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:31.736 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.735+0000 7f0f31ffb640 1 -- 172.21.15.83:0/1844987133 <== mon.1 v1:172.21.15.103:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f0f38004c30 con 0x7f0f3c0704a0 2024-03-25T19:28:31.736 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.735+0000 7f0f31ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:31.736 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.735+0000 7f0f31ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:31.736 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.735+0000 7f0f31ffb640 10 monclient: dump: 2024-03-25T19:28:31.736 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:28:31.736 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:28:31.736 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:31.736 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:31.736 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:31.736 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:28:31.736 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:28:31.737 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:28:31.737 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:28:31.737 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:31.737 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.735+0000 7f0f337fe640 4 set_mon_vals no callback set 2024-03-25T19:28:31.737 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.736+0000 7f0f31ffb640 1 -- 172.21.15.83:0/1844987133 <== mon.1 v1:172.21.15.103:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7f0f3801ac00 con 0x7f0f3c0704a0 2024-03-25T19:28:31.737 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.737+0000 7f0f409a1640 10 monclient: get_auth_request con 0x7f0f2006df70 auth_method 0 2024-03-25T19:28:31.737 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.737+0000 7f0f31ffb640 10 client.4633.objecter ms_handle_connect 0x7f0f2006df70 2024-03-25T19:28:31.744 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.744+0000 7f0f31ffb640 1 -- 172.21.15.83:0/1844987133 <== mon.1 v1:172.21.15.103:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7f0f380547f0 con 0x7f0f3c0704a0 2024-03-25T19:28:31.744 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.744+0000 7f0f31ffb640 10 client.4633.objecter ms_dispatch 0x7f0f3c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:28:31.744 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.744+0000 7f0f31ffb640 3 client.4633.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:28:31.744 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.744+0000 7f0f31ffb640 3 client.4633.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:28:31.744 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.744+0000 7f0f31ffb640 20 client.4633.objecter dump_active .. 0 homeless 2024-03-25T19:28:31.744 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.744+0000 7f0f31ffb640 1 -- 172.21.15.83:0/1844987133 <== mon.1 v1:172.21.15.103:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0f38050130 con 0x7f0f3c0704a0 2024-03-25T19:28:31.744 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.744+0000 7f0f31ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:31.744 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.744+0000 7f0f31ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:31.990 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.990+0000 7f0f43406640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:31.990 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.990+0000 7f0f43406640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:31.990 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.990+0000 7f0f43406640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:28:31.991 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:31.990+0000 7f0f43406640 1 -- 172.21.15.83:0/1844987133 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f3c13db50 con 0x7f0f3c0704a0 2024-03-25T19:28:32.006 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:32.006+0000 7f0f31ffb640 1 -- 172.21.15.83:0/1844987133 <== mon.1 v1:172.21.15.103:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7f0f38052230 con 0x7f0f3c0704a0 2024-03-25T19:28:32.006 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:32.006+0000 7f0f31ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:32.006 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:32.006+0000 7f0f31ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:28:32.006 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:28:32.007 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:28:32.009 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:32.008+0000 7f0f43406640 10 librados: watch_flush enter 2024-03-25T19:28:32.009 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:32.008+0000 7f0f43406640 10 librados: watch_flush exit 2024-03-25T19:28:32.009 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:32.008+0000 7f0f43406640 20 client.4633.objecter shutdown clearing up homeless session... 2024-03-25T19:28:32.009 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:32.008+0000 7f0f43406640 10 client.4633.objecter successfully canceled tick 2024-03-25T19:28:32.009 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:32.008+0000 7f0f43406640 1 -- 172.21.15.83:0/1844987133 >> v1:172.21.15.103:6800/345072327 conn(0x7f0f2006df70 legacy=0x7f0f20070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:32.009 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:32.008+0000 7f0f43406640 10 monclient: shutdown 2024-03-25T19:28:32.009 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:32.008+0000 7f0f43406640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:32.009 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:32.008+0000 7f0f43406640 1 -- 172.21.15.83:0/1844987133 >> v1:172.21.15.103:6789/0 conn(0x7f0f3c0704a0 legacy=0x7f0f3c138310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:32.009 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:32.008+0000 7f0f43406640 1 -- 172.21.15.83:0/1844987133 shutdown_connections 2024-03-25T19:28:32.009 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:32.008+0000 7f0f43406640 1 -- 172.21.15.83:0/1844987133 >> 172.21.15.83:0/1844987133 conn(0x7f0f3c0b5c00 msgr2=0x7f0f3c0b0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:32.009 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:32.008+0000 7f0f43406640 1 -- 172.21.15.83:0/1844987133 shutdown_connections 2024-03-25T19:28:32.009 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:32.009+0000 7f0f43406640 1 -- 172.21.15.83:0/1844987133 wait complete. 2024-03-25T19:28:32.009 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:32.009+0000 7f0f43406640 1 librados: shutdown 2024-03-25T19:28:32.021 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:28:38.828 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:28:38.946 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.946+0000 7fb40d727640 10 monclient: get_monmap_and_config 2024-03-25T19:28:38.946 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.946+0000 7fb40d727640 10 monclient: build_initial_monmap 2024-03-25T19:28:38.946 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.946+0000 7fb40d727640 10 monclient: monmap: 2024-03-25T19:28:38.946 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:28:38.946 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:28:38.946 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:28:38.947393+0000 2024-03-25T19:28:38.946 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:28:38.947393+0000 2024-03-25T19:28:38.946 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:38.947 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:28:38.947 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:28:38.947 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:28:38.947 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:28:38.947 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:38.948 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.947+0000 7fb40d727640 1 Processor -- start 2024-03-25T19:28:38.948 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.947+0000 7fb40d727640 1 -- start start 2024-03-25T19:28:38.948 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.947+0000 7fb40d727640 10 monclient: init 2024-03-25T19:28:38.948 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.947+0000 7fb40d727640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:38.948 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.947+0000 7fb40d727640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:28:38.948 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.947+0000 7fb40d727640 10 monclient(hunting): picked mon.noname-c con 0x7fb4080a98f0 addr v1:172.21.15.103:6789/0 2024-03-25T19:28:38.948 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.947+0000 7fb40d727640 10 monclient(hunting): picked mon.noname-b con 0x7fb40813f710 addr v1:172.21.15.83:6790/0 2024-03-25T19:28:38.948 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.947+0000 7fb40d727640 10 monclient(hunting): picked mon.noname-a con 0x7fb4081430c0 addr v1:172.21.15.83:6789/0 2024-03-25T19:28:38.948 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.947+0000 7fb40d727640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb408142520 con 0x7fb4081430c0 2024-03-25T19:28:38.948 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.947+0000 7fb40d727640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4080b28a0 con 0x7fb4080a98f0 2024-03-25T19:28:38.948 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.947+0000 7fb40d727640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4080b2a30 con 0x7fb40813f710 2024-03-25T19:28:38.949 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.947+0000 7fb40d727640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:38.949 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.947+0000 7fb40d727640 10 monclient(hunting): authenticate will time out at 6576.219539s 2024-03-25T19:28:38.954 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.954+0000 7fb4077fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fb4081430c0 0x7fb40814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:45040/0 (socket says 172.21.15.83:45040) 2024-03-25T19:28:38.954 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.955+0000 7fb4077fe640 1 -- 172.21.15.83:0/788781452 learned_addr learned my addr 172.21.15.83:0/788781452 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:28:38.965 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.963+0000 7fb4067fc640 1 -- 172.21.15.83:0/788781452 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3559665391 0 0) 0x7fb4080b28a0 con 0x7fb4080a98f0 2024-03-25T19:28:38.965 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.963+0000 7fb4067fc640 10 monclient(hunting): _negotiate 2024-03-25T19:28:38.965 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.963+0000 7fb4067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:38.966 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.963+0000 7fb4067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:38.966 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.963+0000 7fb4067fc640 10 monclient(hunting): my global_id is 4636 2024-03-25T19:28:38.966 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.963+0000 7fb4067fc640 1 -- 172.21.15.83:0/788781452 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3f4003520 con 0x7fb4080a98f0 2024-03-25T19:28:38.971 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.971+0000 7fb4067fc640 1 -- 172.21.15.83:0/788781452 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2284903240 0 0) 0x7fb408142520 con 0x7fb4081430c0 2024-03-25T19:28:38.971 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.971+0000 7fb4067fc640 10 monclient(hunting): _negotiate 2024-03-25T19:28:38.971 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.971+0000 7fb4067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:38.971 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.971+0000 7fb4067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:38.971 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.971+0000 7fb4067fc640 10 monclient(hunting): my global_id is 4626 2024-03-25T19:28:38.971 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.971+0000 7fb4067fc640 1 -- 172.21.15.83:0/788781452 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4080b28a0 con 0x7fb4081430c0 2024-03-25T19:28:38.972 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.973+0000 7fb4067fc640 1 -- 172.21.15.83:0/788781452 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1754521298 0 0) 0x7fb4080b2a30 con 0x7fb40813f710 2024-03-25T19:28:38.973 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.973+0000 7fb4067fc640 10 monclient(hunting): _negotiate 2024-03-25T19:28:38.973 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.973+0000 7fb4067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:38.973 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.973+0000 7fb4067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:38.973 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.973+0000 7fb4067fc640 10 monclient(hunting): my global_id is 4640 2024-03-25T19:28:38.973 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.973+0000 7fb4067fc640 1 -- 172.21.15.83:0/788781452 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb408142520 con 0x7fb40813f710 2024-03-25T19:28:38.974 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.973+0000 7fb4067fc640 1 -- 172.21.15.83:0/788781452 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 638483027 0 0) 0x7fb3f4003520 con 0x7fb4080a98f0 2024-03-25T19:28:38.974 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.973+0000 7fb4067fc640 1 -- 172.21.15.83:0/788781452 >> v1:172.21.15.83:6790/0 conn(0x7fb40813f710 legacy=0x7fb4081429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:38.974 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.973+0000 7fb4067fc640 1 -- 172.21.15.83:0/788781452 >> v1:172.21.15.83:6789/0 conn(0x7fb4081430c0 legacy=0x7fb40814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:38.974 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.973+0000 7fb4067fc640 10 monclient: _finish_hunting 0 2024-03-25T19:28:38.974 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.973+0000 7fb4067fc640 1 monclient: found mon.noname-c 2024-03-25T19:28:38.974 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.973+0000 7fb4067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:38.974 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.973+0000 7fb4067fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.103:6789/0 2024-03-25T19:28:38.974 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.973+0000 7fb4067fc640 1 -- 172.21.15.83:0/788781452 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4080b44c0 con 0x7fb4080a98f0 2024-03-25T19:28:38.974 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.973+0000 7fb4067fc640 10 monclient: _finish_auth 0 2024-03-25T19:28:38.974 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.973+0000 7fb4067fc640 10 monclient: _check_auth_tickets 2024-03-25T19:28:38.975 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.973+0000 7fb4067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:38.975 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.973+0000 7fb4067fc640 1 -- 172.21.15.83:0/788781452 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fb3f8001690 con 0x7fb4080a98f0 2024-03-25T19:28:38.975 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.973+0000 7fb4067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:38.975 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.973+0000 7fb4067fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:28:38.975 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.973+0000 7fb4067fc640 10 monclient: dump: 2024-03-25T19:28:38.975 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:28:38.975 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:28:38.975 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:38.975 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:38.975 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:38.975 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:28:38.975 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:28:38.975 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:28:38.975 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:28:38.976 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:38.976 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.973+0000 7fb40d727640 5 monclient: authenticate success, global_id 4636 2024-03-25T19:28:38.976 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.973+0000 7fb40d727640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:38.986 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.987+0000 7fb4067fc640 1 -- 172.21.15.83:0/788781452 <== mon.1 v1:172.21.15.103:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb3f80017f0 con 0x7fb4080a98f0 2024-03-25T19:28:38.986 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.987+0000 7fb4067fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:38.987 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.987+0000 7fb4067fc640 1 -- 172.21.15.83:0/788781452 <== mon.1 v1:172.21.15.103:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fb3f8001c00 con 0x7fb4080a98f0 2024-03-25T19:28:38.987 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.987+0000 7fb40d727640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:38.987 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.987+0000 7fb40d727640 4 set_mon_vals no callback set 2024-03-25T19:28:38.987 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.987+0000 7fb4067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:38.987 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.987+0000 7fb4067fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:38.987 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.987+0000 7fb4067fc640 10 monclient: dump: 2024-03-25T19:28:38.987 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:28:38.987 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:28:38.987 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:38.987 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:38.987 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:38.987 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:28:38.987 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:28:38.987 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:28:38.987 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:28:38.987 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:38.989 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.988+0000 7fb40d727640 10 monclient: shutdown 2024-03-25T19:28:38.989 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.988+0000 7fb40d727640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:38.989 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.988+0000 7fb40d727640 1 -- 172.21.15.83:0/788781452 >> v1:172.21.15.103:6789/0 conn(0x7fb4080a98f0 legacy=0x7fb4080a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:38.989 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.988+0000 7fb40d727640 1 -- 172.21.15.83:0/788781452 shutdown_connections 2024-03-25T19:28:38.989 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.988+0000 7fb40d727640 1 -- 172.21.15.83:0/788781452 >> 172.21.15.83:0/788781452 conn(0x7fb4080b7ea0 msgr2=0x7fb4080b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:38.989 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.988+0000 7fb40d727640 1 -- 172.21.15.83:0/788781452 shutdown_connections 2024-03-25T19:28:38.989 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.988+0000 7fb40d727640 1 -- 172.21.15.83:0/788781452 wait complete. 2024-03-25T19:28:38.989 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.989+0000 7fb40d727640 10 monclient: build_initial_monmap 2024-03-25T19:28:38.989 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.989+0000 7fb40d727640 10 monclient: monmap: 2024-03-25T19:28:38.989 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:28:38.989 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:38.989 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:28:38.989 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:28:38.989 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:38.989 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:28:38.990 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:28:38.990 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:28:38.990 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:28:38.990 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:38.990 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.989+0000 7fb40d727640 1 librados: starting msgr at 2024-03-25T19:28:38.990 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.989+0000 7fb40d727640 1 librados: starting objecter 2024-03-25T19:28:38.990 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.989+0000 7fb40d727640 1 Processor -- start 2024-03-25T19:28:38.991 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.989+0000 7fb40d727640 1 -- start start 2024-03-25T19:28:38.991 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.989+0000 7fb40d727640 1 librados: setting wanted keys 2024-03-25T19:28:38.991 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.989+0000 7fb40d727640 1 librados: calling monclient init 2024-03-25T19:28:38.991 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.989+0000 7fb40d727640 10 monclient: init 2024-03-25T19:28:38.991 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.990+0000 7fb40d727640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:38.991 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.990+0000 7fb40d727640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:28:38.991 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.990+0000 7fb40d727640 10 monclient(hunting): picked mon.noname-c con 0x7fb4080a98f0 addr v1:172.21.15.83:6790/0 2024-03-25T19:28:38.991 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.990+0000 7fb40d727640 10 monclient(hunting): picked mon.noname-a con 0x7fb40813f710 addr v1:172.21.15.83:6789/0 2024-03-25T19:28:38.992 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.990+0000 7fb40d727640 10 monclient(hunting): picked mon.noname-b con 0x7fb4081430c0 addr v1:172.21.15.103:6789/0 2024-03-25T19:28:38.992 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.990+0000 7fb40d727640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb408139a40 con 0x7fb40813f710 2024-03-25T19:28:38.992 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.990+0000 7fb40d727640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb408133f20 con 0x7fb4081430c0 2024-03-25T19:28:38.992 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.990+0000 7fb40d727640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4080ab950 con 0x7fb4080a98f0 2024-03-25T19:28:38.992 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.990+0000 7fb40d727640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:38.992 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:38.990+0000 7fb40d727640 10 monclient(hunting): authenticate will time out at 6576.262336s 2024-03-25T19:28:39.003 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.004+0000 7fb4077fe640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fb4081430c0 0x7fb408133810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:51120/0 (socket says 172.21.15.83:51120) 2024-03-25T19:28:39.003 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.004+0000 7fb4077fe640 1 -- 172.21.15.83:0/1436055593 learned_addr learned my addr 172.21.15.83:0/1436055593 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:28:39.004 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.004+0000 7fb3fffff640 10 client.?.objecter ms_handle_connect 0x7fb4081430c0 2024-03-25T19:28:39.004 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.004+0000 7fb3fffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:39.005 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.006+0000 7fb3fffff640 10 client.?.objecter ms_handle_connect 0x7fb4080a98f0 2024-03-25T19:28:39.005 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.006+0000 7fb3fffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:39.013 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.013+0000 7fb3fffff640 1 -- 172.21.15.83:0/1436055593 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3517447183 0 0) 0x7fb408133f20 con 0x7fb4081430c0 2024-03-25T19:28:39.013 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.014+0000 7fb3fffff640 10 monclient(hunting): _negotiate 2024-03-25T19:28:39.013 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.014+0000 7fb3fffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:39.013 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.014+0000 7fb3fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:39.013 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.014+0000 7fb3fffff640 10 monclient(hunting): my global_id is 4639 2024-03-25T19:28:39.013 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.014+0000 7fb3fffff640 1 -- 172.21.15.83:0/1436055593 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3e8004eb0 con 0x7fb4081430c0 2024-03-25T19:28:39.018 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.019+0000 7fb3fffff640 10 client.?.objecter ms_handle_connect 0x7fb40813f710 2024-03-25T19:28:39.018 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.019+0000 7fb3fffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:39.021 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.022+0000 7fb3fffff640 1 -- 172.21.15.83:0/1436055593 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 536857002 0 0) 0x7fb4080ab950 con 0x7fb4080a98f0 2024-03-25T19:28:39.021 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.022+0000 7fb3fffff640 10 monclient(hunting): _negotiate 2024-03-25T19:28:39.021 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.022+0000 7fb3fffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:39.021 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.022+0000 7fb3fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:39.022 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.022+0000 7fb3fffff640 10 monclient(hunting): my global_id is 4643 2024-03-25T19:28:39.022 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.022+0000 7fb3fffff640 1 -- 172.21.15.83:0/1436055593 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb408133f20 con 0x7fb4080a98f0 2024-03-25T19:28:39.030 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.031+0000 7fb3fffff640 1 -- 172.21.15.83:0/1436055593 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 549505455 0 0) 0x7fb3e8004eb0 con 0x7fb4081430c0 2024-03-25T19:28:39.031 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.031+0000 7fb3fffff640 1 -- 172.21.15.83:0/1436055593 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb4080ab950 con 0x7fb4081430c0 2024-03-25T19:28:39.031 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.031+0000 7fb3fffff640 1 -- 172.21.15.83:0/1436055593 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fb400001c70 con 0x7fb4081430c0 2024-03-25T19:28:39.031 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.031+0000 7fb3fffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:39.031 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.031+0000 7fb3fffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:28:39.031 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.031+0000 7fb3fffff640 10 monclient(hunting): dump: 2024-03-25T19:28:39.031 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:28:39.031 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:28:39.031 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:39.031 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:39.031 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:39.031 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:28:39.031 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:28:39.031 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:28:39.031 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:28:39.031 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:39.032 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.031+0000 7fb3fffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:39.033 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.033+0000 7fb3fffff640 1 -- 172.21.15.83:0/1436055593 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 545100582 0 0) 0x7fb408139a40 con 0x7fb40813f710 2024-03-25T19:28:39.033 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.033+0000 7fb3fffff640 10 monclient(hunting): _negotiate 2024-03-25T19:28:39.033 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.033+0000 7fb3fffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:39.033 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.033+0000 7fb3fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:39.033 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.033+0000 7fb3fffff640 10 monclient(hunting): my global_id is 4629 2024-03-25T19:28:39.033 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.033+0000 7fb3fffff640 1 -- 172.21.15.83:0/1436055593 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3e8004eb0 con 0x7fb40813f710 2024-03-25T19:28:39.039 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.039+0000 7fb3fffff640 1 -- 172.21.15.83:0/1436055593 <== mon.1 v1:172.21.15.103:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3087666546 0 0) 0x7fb4080ab950 con 0x7fb4081430c0 2024-03-25T19:28:39.039 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.039+0000 7fb3fffff640 1 -- 172.21.15.83:0/1436055593 >> v1:172.21.15.83:6790/0 conn(0x7fb4080a98f0 legacy=0x7fb408138c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:39.039 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.039+0000 7fb3fffff640 1 -- 172.21.15.83:0/1436055593 >> v1:172.21.15.83:6789/0 conn(0x7fb40813f710 legacy=0x7fb408133100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:39.039 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.039+0000 7fb3fffff640 10 monclient: _finish_hunting 0 2024-03-25T19:28:39.039 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.039+0000 7fb3fffff640 1 monclient: found mon.b 2024-03-25T19:28:39.039 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.039+0000 7fb3fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:39.039 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.039+0000 7fb3fffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:28:39.039 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.039+0000 7fb3fffff640 1 -- 172.21.15.83:0/1436055593 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4080abc30 con 0x7fb4081430c0 2024-03-25T19:28:39.040 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.039+0000 7fb3fffff640 10 monclient: _finish_auth 0 2024-03-25T19:28:39.040 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.039+0000 7fb3fffff640 10 monclient: _check_auth_tickets 2024-03-25T19:28:39.040 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.039+0000 7fb3fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:39.040 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.039+0000 7fb40d727640 5 monclient: authenticate success, global_id 4639 2024-03-25T19:28:39.040 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.039+0000 7fb40d727640 10 monclient: _renew_subs 2024-03-25T19:28:39.040 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.039+0000 7fb40d727640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:28:39.040 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.039+0000 7fb40d727640 1 -- 172.21.15.83:0/1436055593 --> v1:172.21.15.103:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4080abf10 con 0x7fb4081430c0 2024-03-25T19:28:39.040 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.039+0000 7fb40d727640 10 client.4639.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:39.040 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.039+0000 7fb40d727640 10 monclient: _renew_subs 2024-03-25T19:28:39.040 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.039+0000 7fb40d727640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:28:39.040 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.039+0000 7fb40d727640 1 -- 172.21.15.83:0/1436055593 --> v1:172.21.15.103:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb408139370 con 0x7fb4081430c0 2024-03-25T19:28:39.040 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.039+0000 7fb40d727640 1 librados: init done 2024-03-25T19:28:39.041 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.040+0000 7fb40d727640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:39.041 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.040+0000 7fb40d727640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:39.041 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.040+0000 7fb40d727640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:28:39.041 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.040+0000 7fb40d727640 1 -- 172.21.15.83:0/1436055593 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4080627b0 con 0x7fb4081430c0 2024-03-25T19:28:39.318 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.318+0000 7fb3fffff640 1 -- 172.21.15.83:0/1436055593 <== mon.1 v1:172.21.15.103:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb400004b60 con 0x7fb4081430c0 2024-03-25T19:28:39.318 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.318+0000 7fb3fffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:39.318 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.318+0000 7fb3fffff640 1 -- 172.21.15.83:0/1436055593 <== mon.1 v1:172.21.15.103:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fb400004e70 con 0x7fb4081430c0 2024-03-25T19:28:39.318 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.318+0000 7fb3fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:39.318 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.318+0000 7fb3fffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:39.318 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.318+0000 7fb3fffff640 10 monclient: dump: 2024-03-25T19:28:39.318 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:28:39.319 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:28:39.319 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:39.319 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:39.319 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:39.319 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:28:39.319 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:28:39.319 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:28:39.319 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:28:39.319 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:39.319 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.318+0000 7fb405ffb640 4 set_mon_vals no callback set 2024-03-25T19:28:39.320 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.319+0000 7fb3fffff640 1 -- 172.21.15.83:0/1436055593 <== mon.1 v1:172.21.15.103:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7fb40001ae40 con 0x7fb4081430c0 2024-03-25T19:28:39.320 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.320+0000 7fb406ffd640 10 monclient: get_auth_request con 0x7fb3e806de00 auth_method 0 2024-03-25T19:28:39.320 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.320+0000 7fb3fffff640 10 client.4639.objecter ms_handle_connect 0x7fb3e806de00 2024-03-25T19:28:39.323 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.324+0000 7fb3fffff640 1 -- 172.21.15.83:0/1436055593 <== mon.1 v1:172.21.15.103:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7fb400054b00 con 0x7fb4081430c0 2024-03-25T19:28:39.323 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.324+0000 7fb3fffff640 10 client.4639.objecter ms_dispatch 0x7fb408000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:28:39.323 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.324+0000 7fb3fffff640 3 client.4639.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:28:39.323 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.324+0000 7fb3fffff640 3 client.4639.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:28:39.323 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.324+0000 7fb3fffff640 20 client.4639.objecter dump_active .. 0 homeless 2024-03-25T19:28:39.325 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.325+0000 7fb3fffff640 1 -- 172.21.15.83:0/1436055593 <== mon.1 v1:172.21.15.103:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb400050440 con 0x7fb4081430c0 2024-03-25T19:28:39.325 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.325+0000 7fb3fffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:39.325 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.325+0000 7fb3fffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:39.537 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.537+0000 7fb40d727640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:39.537 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.537+0000 7fb40d727640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:39.537 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.537+0000 7fb40d727640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:28:39.537 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.537+0000 7fb40d727640 1 -- 172.21.15.83:0/1436055593 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb4080b7180 con 0x7fb4081430c0 2024-03-25T19:28:39.593 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.594+0000 7fb3fffff640 1 -- 172.21.15.83:0/1436055593 <== mon.1 v1:172.21.15.103:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7fb400052540 con 0x7fb4081430c0 2024-03-25T19:28:39.594 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.594+0000 7fb3fffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:39.594 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.594+0000 7fb3fffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:28:39.594 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:28:39.594 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:28:39.597 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.596+0000 7fb40d727640 10 librados: watch_flush enter 2024-03-25T19:28:39.597 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.596+0000 7fb40d727640 10 librados: watch_flush exit 2024-03-25T19:28:39.597 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.596+0000 7fb40d727640 20 client.4639.objecter shutdown clearing up homeless session... 2024-03-25T19:28:39.597 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.596+0000 7fb40d727640 10 client.4639.objecter successfully canceled tick 2024-03-25T19:28:39.597 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.596+0000 7fb40d727640 1 -- 172.21.15.83:0/1436055593 >> v1:172.21.15.103:6800/345072327 conn(0x7fb3e806de00 legacy=0x7fb3e80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:39.597 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.596+0000 7fb40d727640 10 monclient: shutdown 2024-03-25T19:28:39.597 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.596+0000 7fb40d727640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:39.597 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.596+0000 7fb40d727640 1 -- 172.21.15.83:0/1436055593 >> v1:172.21.15.103:6789/0 conn(0x7fb4081430c0 legacy=0x7fb408133810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:39.597 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.596+0000 7fb40d727640 1 -- 172.21.15.83:0/1436055593 shutdown_connections 2024-03-25T19:28:39.597 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.596+0000 7fb40d727640 1 -- 172.21.15.83:0/1436055593 >> 172.21.15.83:0/1436055593 conn(0x7fb4080b7ea0 msgr2=0x7fb408074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:39.597 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.596+0000 7fb40d727640 1 -- 172.21.15.83:0/1436055593 shutdown_connections 2024-03-25T19:28:39.597 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.596+0000 7fb40d727640 1 -- 172.21.15.83:0/1436055593 wait complete. 2024-03-25T19:28:39.597 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:39.597+0000 7fb40d727640 1 librados: shutdown 2024-03-25T19:28:39.606 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:28:46.413 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:28:46.528 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.529+0000 7f081c64e640 10 monclient: get_monmap_and_config 2024-03-25T19:28:46.529 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.529+0000 7f081c64e640 10 monclient: build_initial_monmap 2024-03-25T19:28:46.529 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.529+0000 7f081c64e640 10 monclient: monmap: 2024-03-25T19:28:46.529 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:28:46.529 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:28:46.529 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:28:46.529972+0000 2024-03-25T19:28:46.529 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:28:46.529972+0000 2024-03-25T19:28:46.529 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:46.529 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:28:46.529 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:28:46.529 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:28:46.529 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:28:46.530 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:46.531 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.529+0000 7f081c64e640 1 Processor -- start 2024-03-25T19:28:46.531 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.529+0000 7f081c64e640 1 -- start start 2024-03-25T19:28:46.531 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.529+0000 7f081c64e640 10 monclient: init 2024-03-25T19:28:46.531 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.530+0000 7f081c64e640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:46.531 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.530+0000 7f081c64e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:28:46.532 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.530+0000 7f081c64e640 10 monclient(hunting): picked mon.noname-b con 0x7f08140a98f0 addr v1:172.21.15.83:6790/0 2024-03-25T19:28:46.532 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.531+0000 7f081c64e640 10 monclient(hunting): picked mon.noname-c con 0x7f081413f710 addr v1:172.21.15.103:6789/0 2024-03-25T19:28:46.532 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.531+0000 7f081c64e640 10 monclient(hunting): picked mon.noname-a con 0x7f08141430c0 addr v1:172.21.15.83:6789/0 2024-03-25T19:28:46.532 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.531+0000 7f081c64e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0814142520 con 0x7f08141430c0 2024-03-25T19:28:46.532 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.531+0000 7f081c64e640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08140b28a0 con 0x7f081413f710 2024-03-25T19:28:46.532 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.531+0000 7f081c64e640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08140b2a30 con 0x7f08140a98f0 2024-03-25T19:28:46.532 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.531+0000 7f081c64e640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:46.532 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.531+0000 7f081c64e640 10 monclient(hunting): authenticate will time out at 6583.803160s 2024-03-25T19:28:46.538 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.539+0000 7f0819be9640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f081413f710 0x7f08141429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:60098/0 (socket says 172.21.15.83:60098) 2024-03-25T19:28:46.538 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.539+0000 7f0819be9640 1 -- 172.21.15.83:0/3663966771 learned_addr learned my addr 172.21.15.83:0/3663966771 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:28:46.560 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.560+0000 7f08193e8640 1 -- 172.21.15.83:0/3663966771 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1125083222 0 0) 0x7f0814142520 con 0x7f08141430c0 2024-03-25T19:28:46.560 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.560+0000 7f08193e8640 10 monclient(hunting): _negotiate 2024-03-25T19:28:46.560 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.560+0000 7f08193e8640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:46.560 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.560+0000 7f08193e8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:46.560 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.560+0000 7f08193e8640 10 monclient(hunting): my global_id is 4632 2024-03-25T19:28:46.560 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.560+0000 7f08193e8640 1 -- 172.21.15.83:0/3663966771 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0800003520 con 0x7f08141430c0 2024-03-25T19:28:46.561 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.561+0000 7f08193e8640 1 -- 172.21.15.83:0/3663966771 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 450694805 0 0) 0x7f08140b28a0 con 0x7f081413f710 2024-03-25T19:28:46.561 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.561+0000 7f08193e8640 10 monclient(hunting): _negotiate 2024-03-25T19:28:46.561 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.561+0000 7f08193e8640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:46.561 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.561+0000 7f08193e8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:46.561 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.561+0000 7f08193e8640 10 monclient(hunting): my global_id is 4642 2024-03-25T19:28:46.561 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.561+0000 7f08193e8640 1 -- 172.21.15.83:0/3663966771 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0814142520 con 0x7f081413f710 2024-03-25T19:28:46.562 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.562+0000 7f08193e8640 1 -- 172.21.15.83:0/3663966771 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2702665411 0 0) 0x7f08140b2a30 con 0x7f08140a98f0 2024-03-25T19:28:46.562 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.562+0000 7f08193e8640 10 monclient(hunting): _negotiate 2024-03-25T19:28:46.562 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.562+0000 7f08193e8640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:46.562 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.562+0000 7f08193e8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:46.562 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.562+0000 7f08193e8640 10 monclient(hunting): my global_id is 4646 2024-03-25T19:28:46.562 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.562+0000 7f08193e8640 1 -- 172.21.15.83:0/3663966771 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08140b28a0 con 0x7f08140a98f0 2024-03-25T19:28:46.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.580+0000 7f08193e8640 1 -- 172.21.15.83:0/3663966771 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1990128720 0 0) 0x7f08140b28a0 con 0x7f08140a98f0 2024-03-25T19:28:46.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.580+0000 7f08193e8640 1 -- 172.21.15.83:0/3663966771 >> v1:172.21.15.103:6789/0 conn(0x7f081413f710 legacy=0x7f08141429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:46.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.580+0000 7f08193e8640 1 -- 172.21.15.83:0/3663966771 >> v1:172.21.15.83:6789/0 conn(0x7f08141430c0 legacy=0x7f081414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:46.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.580+0000 7f08193e8640 10 monclient: _finish_hunting 0 2024-03-25T19:28:46.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.580+0000 7f08193e8640 1 monclient: found mon.noname-b 2024-03-25T19:28:46.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.580+0000 7f08193e8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:46.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.580+0000 7f08193e8640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.83:6790/0 2024-03-25T19:28:46.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.580+0000 7f08193e8640 1 -- 172.21.15.83:0/3663966771 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08140b44c0 con 0x7f08140a98f0 2024-03-25T19:28:46.581 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.580+0000 7f08193e8640 10 monclient: _finish_auth 0 2024-03-25T19:28:46.581 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.580+0000 7f08193e8640 10 monclient: _check_auth_tickets 2024-03-25T19:28:46.581 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.580+0000 7f08193e8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:46.581 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.580+0000 7f08193e8640 1 -- 172.21.15.83:0/3663966771 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f08100015b0 con 0x7f08140a98f0 2024-03-25T19:28:46.581 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.580+0000 7f081c64e640 5 monclient: authenticate success, global_id 4646 2024-03-25T19:28:46.581 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.580+0000 7f081c64e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:46.581 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.580+0000 7f08193e8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:46.581 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.580+0000 7f08193e8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:28:46.581 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.580+0000 7f08193e8640 10 monclient: dump: 2024-03-25T19:28:46.581 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:28:46.581 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:28:46.581 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:46.581 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:46.581 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:46.581 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:28:46.581 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:28:46.582 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:28:46.582 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:28:46.582 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:46.582 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.580+0000 7f081c64e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:46.601 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.601+0000 7f08193e8640 1 -- 172.21.15.83:0/3663966771 <== mon.2 v1:172.21.15.83:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0810001710 con 0x7f08140a98f0 2024-03-25T19:28:46.601 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.601+0000 7f08193e8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:46.601 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.601+0000 7f08193e8640 1 -- 172.21.15.83:0/3663966771 <== mon.2 v1:172.21.15.83:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f0810001b60 con 0x7f08140a98f0 2024-03-25T19:28:46.601 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.601+0000 7f08193e8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:46.601 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.601+0000 7f08193e8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:28:46.601 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.601+0000 7f08193e8640 10 monclient: dump: 2024-03-25T19:28:46.601 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:28:46.601 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:28:46.601 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:46.601 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:46.601 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:46.601 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:28:46.602 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:28:46.602 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:28:46.602 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:28:46.602 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:46.603 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.601+0000 7f081c64e640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:46.603 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.601+0000 7f081c64e640 4 set_mon_vals no callback set 2024-03-25T19:28:46.604 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.602+0000 7f081c64e640 10 monclient: shutdown 2024-03-25T19:28:46.604 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.602+0000 7f081c64e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:46.604 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.602+0000 7f081c64e640 1 -- 172.21.15.83:0/3663966771 >> v1:172.21.15.83:6790/0 conn(0x7f08140a98f0 legacy=0x7f08140a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:46.604 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.602+0000 7f081c64e640 1 -- 172.21.15.83:0/3663966771 shutdown_connections 2024-03-25T19:28:46.604 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.602+0000 7f081c64e640 1 -- 172.21.15.83:0/3663966771 >> 172.21.15.83:0/3663966771 conn(0x7f08140b7ea0 msgr2=0x7f08140b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:46.604 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.602+0000 7f081c64e640 1 -- 172.21.15.83:0/3663966771 shutdown_connections 2024-03-25T19:28:46.604 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.602+0000 7f081c64e640 1 -- 172.21.15.83:0/3663966771 wait complete. 2024-03-25T19:28:46.604 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.603+0000 7f081c64e640 10 monclient: build_initial_monmap 2024-03-25T19:28:46.604 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.603+0000 7f081c64e640 10 monclient: monmap: 2024-03-25T19:28:46.604 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:28:46.604 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:46.604 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:28:46.604 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:28:46.604 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:46.605 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:28:46.605 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:28:46.605 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:28:46.605 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:28:46.605 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:46.605 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.604+0000 7f081c64e640 1 librados: starting msgr at 2024-03-25T19:28:46.605 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.604+0000 7f081c64e640 1 librados: starting objecter 2024-03-25T19:28:46.605 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.604+0000 7f081c64e640 1 Processor -- start 2024-03-25T19:28:46.605 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.604+0000 7f081c64e640 1 -- start start 2024-03-25T19:28:46.605 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.604+0000 7f081c64e640 1 librados: setting wanted keys 2024-03-25T19:28:46.605 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.604+0000 7f081c64e640 1 librados: calling monclient init 2024-03-25T19:28:46.605 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.604+0000 7f081c64e640 10 monclient: init 2024-03-25T19:28:46.606 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.604+0000 7f081c64e640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:46.606 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.604+0000 7f081c64e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:28:46.607 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.604+0000 7f081c64e640 10 monclient(hunting): picked mon.noname-b con 0x7f08140a98f0 addr v1:172.21.15.103:6789/0 2024-03-25T19:28:46.607 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.604+0000 7f081c64e640 10 monclient(hunting): picked mon.noname-a con 0x7f081413f710 addr v1:172.21.15.83:6789/0 2024-03-25T19:28:46.607 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.604+0000 7f081c64e640 10 monclient(hunting): picked mon.noname-c con 0x7f08141430c0 addr v1:172.21.15.83:6790/0 2024-03-25T19:28:46.607 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.604+0000 7f081c64e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08141d78a0 con 0x7f081413f710 2024-03-25T19:28:46.607 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.604+0000 7f081c64e640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08141d51f0 con 0x7f08140a98f0 2024-03-25T19:28:46.607 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.604+0000 7f081c64e640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08141d8440 con 0x7f08141430c0 2024-03-25T19:28:46.607 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.604+0000 7f081c64e640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:46.607 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.604+0000 7f081c64e640 10 monclient(hunting): authenticate will time out at 6583.876680s 2024-03-25T19:28:46.612 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.613+0000 7f081abeb640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f08141430c0 0x7f08141e1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:33126/0 (socket says 172.21.15.83:33126) 2024-03-25T19:28:46.612 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.613+0000 7f081abeb640 1 -- 172.21.15.83:0/2580613895 learned_addr learned my addr 172.21.15.83:0/2580613895 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:28:46.614 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.615+0000 7f080affd640 10 client.?.objecter ms_handle_connect 0x7f08140a98f0 2024-03-25T19:28:46.614 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.615+0000 7f080affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:46.617 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.617+0000 7f080affd640 10 client.?.objecter ms_handle_connect 0x7f08141430c0 2024-03-25T19:28:46.617 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.617+0000 7f080affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:46.624 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.625+0000 7f080affd640 1 -- 172.21.15.83:0/2580613895 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1230873879 0 0) 0x7f08141d8440 con 0x7f08141430c0 2024-03-25T19:28:46.625 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.625+0000 7f080affd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:46.625 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.625+0000 7f080affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:46.625 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.625+0000 7f080affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:46.625 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.625+0000 7f080affd640 10 monclient(hunting): my global_id is 4649 2024-03-25T19:28:46.625 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.625+0000 7f080affd640 1 -- 172.21.15.83:0/2580613895 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07f8004eb0 con 0x7f08141430c0 2024-03-25T19:28:46.625 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.625+0000 7f080affd640 1 -- 172.21.15.83:0/2580613895 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2162194790 0 0) 0x7f08141d51f0 con 0x7f08140a98f0 2024-03-25T19:28:46.625 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.625+0000 7f080affd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:46.626 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.625+0000 7f080affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:46.626 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.625+0000 7f080affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:46.626 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.625+0000 7f080affd640 10 monclient(hunting): my global_id is 4645 2024-03-25T19:28:46.626 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.625+0000 7f080affd640 1 -- 172.21.15.83:0/2580613895 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08141d8440 con 0x7f08140a98f0 2024-03-25T19:28:46.632 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.632+0000 7f080affd640 10 client.?.objecter ms_handle_connect 0x7f081413f710 2024-03-25T19:28:46.632 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.632+0000 7f080affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:46.642 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.642+0000 7f080affd640 1 -- 172.21.15.83:0/2580613895 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2856396042 0 0) 0x7f08141d78a0 con 0x7f081413f710 2024-03-25T19:28:46.642 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.642+0000 7f080affd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:46.642 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.642+0000 7f080affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:46.642 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.642+0000 7f080affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:46.643 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.643+0000 7f080affd640 10 monclient(hunting): my global_id is 4635 2024-03-25T19:28:46.643 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.643+0000 7f080affd640 1 -- 172.21.15.83:0/2580613895 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08141d51f0 con 0x7f081413f710 2024-03-25T19:28:46.643 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.644+0000 7f080affd640 1 -- 172.21.15.83:0/2580613895 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2089471728 0 0) 0x7f08141d8440 con 0x7f08140a98f0 2024-03-25T19:28:46.644 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.644+0000 7f080affd640 1 -- 172.21.15.83:0/2580613895 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08141d78a0 con 0x7f08140a98f0 2024-03-25T19:28:46.644 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.644+0000 7f080affd640 1 -- 172.21.15.83:0/2580613895 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f0810001a20 con 0x7f08140a98f0 2024-03-25T19:28:46.644 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.644+0000 7f080affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:46.644 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.644+0000 7f080affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:28:46.644 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.644+0000 7f080affd640 10 monclient(hunting): dump: 2024-03-25T19:28:46.644 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:28:46.644 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:28:46.644 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:46.644 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:46.644 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:46.644 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:28:46.645 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:28:46.645 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:28:46.645 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:28:46.645 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:46.645 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.644+0000 7f080affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:46.651 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.651+0000 7f080affd640 1 -- 172.21.15.83:0/2580613895 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2250659330 0 0) 0x7f07f8004eb0 con 0x7f08141430c0 2024-03-25T19:28:46.651 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.651+0000 7f080affd640 1 -- 172.21.15.83:0/2580613895 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08141d8440 con 0x7f08141430c0 2024-03-25T19:28:46.651 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.651+0000 7f080affd640 1 -- 172.21.15.83:0/2580613895 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f080c001c30 con 0x7f08141430c0 2024-03-25T19:28:46.651 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.651+0000 7f080affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:46.651 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.651+0000 7f080affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:28:46.651 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.651+0000 7f080affd640 10 monclient(hunting): dump: 2024-03-25T19:28:46.651 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:28:46.651 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:28:46.651 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:46.651 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:46.651 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:46.651 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:28:46.652 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:28:46.652 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:28:46.652 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:28:46.652 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:46.661 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.661+0000 7f080affd640 1 -- 172.21.15.83:0/2580613895 <== mon.1 v1:172.21.15.103:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1699206766 0 0) 0x7f08141d78a0 con 0x7f08140a98f0 2024-03-25T19:28:46.661 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.661+0000 7f080affd640 1 -- 172.21.15.83:0/2580613895 >> v1:172.21.15.83:6790/0 conn(0x7f08141430c0 legacy=0x7f08141e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:46.661 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.661+0000 7f080affd640 1 -- 172.21.15.83:0/2580613895 >> v1:172.21.15.83:6789/0 conn(0x7f081413f710 legacy=0x7f08141385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:46.662 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.661+0000 7f080affd640 10 monclient: _finish_hunting 0 2024-03-25T19:28:46.662 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.661+0000 7f080affd640 1 monclient: found mon.b 2024-03-25T19:28:46.662 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.661+0000 7f080affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:46.662 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.661+0000 7f080affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:28:46.662 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.661+0000 7f080affd640 1 -- 172.21.15.83:0/2580613895 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0814138cb0 con 0x7f08140a98f0 2024-03-25T19:28:46.662 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.661+0000 7f080affd640 10 monclient: _finish_auth 0 2024-03-25T19:28:46.662 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.661+0000 7f080affd640 10 monclient: _check_auth_tickets 2024-03-25T19:28:46.662 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.661+0000 7f080affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:46.662 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.661+0000 7f081c64e640 5 monclient: authenticate success, global_id 4645 2024-03-25T19:28:46.662 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.661+0000 7f081c64e640 10 monclient: _renew_subs 2024-03-25T19:28:46.662 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.661+0000 7f081c64e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:28:46.662 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.661+0000 7f081c64e640 1 -- 172.21.15.83:0/2580613895 --> v1:172.21.15.103:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0814137890 con 0x7f08140a98f0 2024-03-25T19:28:46.663 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.661+0000 7f081c64e640 10 client.4645.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:46.663 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.661+0000 7f081c64e640 10 monclient: _renew_subs 2024-03-25T19:28:46.663 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.661+0000 7f081c64e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:28:46.663 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.661+0000 7f081c64e640 1 -- 172.21.15.83:0/2580613895 --> v1:172.21.15.103:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f08141e3550 con 0x7f08140a98f0 2024-03-25T19:28:46.663 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.661+0000 7f081c64e640 1 librados: init done 2024-03-25T19:28:46.663 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.662+0000 7f081c64e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:46.663 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.662+0000 7f081c64e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:46.663 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.662+0000 7f081c64e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:28:46.664 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.662+0000 7f081c64e640 1 -- 172.21.15.83:0/2580613895 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08141d7600 con 0x7f08140a98f0 2024-03-25T19:28:46.953 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.953+0000 7f080affd640 1 -- 172.21.15.83:0/2580613895 <== mon.1 v1:172.21.15.103:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f08100038a0 con 0x7f08140a98f0 2024-03-25T19:28:46.953 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.953+0000 7f080affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:46.954 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.953+0000 7f0818be7640 4 set_mon_vals no callback set 2024-03-25T19:28:46.954 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.953+0000 7f080affd640 1 -- 172.21.15.83:0/2580613895 <== mon.1 v1:172.21.15.103:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f0810001c20 con 0x7f08140a98f0 2024-03-25T19:28:46.954 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.953+0000 7f080affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:46.954 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.953+0000 7f080affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:46.954 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.953+0000 7f080affd640 10 monclient: dump: 2024-03-25T19:28:46.954 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:28:46.954 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:28:46.954 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:46.954 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:46.954 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:46.954 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:28:46.954 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:28:46.954 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:28:46.955 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:28:46.955 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:46.955 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.954+0000 7f080affd640 1 -- 172.21.15.83:0/2580613895 <== mon.1 v1:172.21.15.103:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7f0810019a20 con 0x7f08140a98f0 2024-03-25T19:28:46.955 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.955+0000 7f0819be9640 10 monclient: get_auth_request con 0x7f07f806df70 auth_method 0 2024-03-25T19:28:46.955 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.955+0000 7f080affd640 10 client.4645.objecter ms_handle_connect 0x7f07f806df70 2024-03-25T19:28:46.958 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.958+0000 7f080affd640 1 -- 172.21.15.83:0/2580613895 <== mon.1 v1:172.21.15.103:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7f0810053640 con 0x7f08140a98f0 2024-03-25T19:28:46.958 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.958+0000 7f080affd640 10 client.4645.objecter ms_dispatch 0x7f0814000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:28:46.958 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.958+0000 7f080affd640 3 client.4645.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:28:46.958 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.958+0000 7f080affd640 3 client.4645.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:28:46.959 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.958+0000 7f080affd640 20 client.4645.objecter dump_active .. 0 homeless 2024-03-25T19:28:46.959 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.958+0000 7f080affd640 1 -- 172.21.15.83:0/2580613895 <== mon.1 v1:172.21.15.103:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f081004f150 con 0x7f08140a98f0 2024-03-25T19:28:46.959 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.958+0000 7f080affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:46.959 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:46.958+0000 7f080affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:47.207 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:47.207+0000 7f081c64e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:47.207 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:47.207+0000 7f081c64e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:47.207 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:47.207+0000 7f081c64e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:28:47.207 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:47.207+0000 7f081c64e640 1 -- 172.21.15.83:0/2580613895 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f08140627b0 con 0x7f08140a98f0 2024-03-25T19:28:47.281 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:47.282+0000 7f080affd640 1 -- 172.21.15.83:0/2580613895 <== mon.1 v1:172.21.15.103:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7f0810051080 con 0x7f08140a98f0 2024-03-25T19:28:47.281 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:47.282+0000 7f080affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:47.281 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:47.282+0000 7f080affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:28:47.282 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:28:47.282 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:28:47.284 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:47.284+0000 7f081c64e640 10 librados: watch_flush enter 2024-03-25T19:28:47.284 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:47.284+0000 7f081c64e640 10 librados: watch_flush exit 2024-03-25T19:28:47.284 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:47.284+0000 7f081c64e640 20 client.4645.objecter shutdown clearing up homeless session... 2024-03-25T19:28:47.284 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:47.284+0000 7f081c64e640 10 client.4645.objecter successfully canceled tick 2024-03-25T19:28:47.284 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:47.284+0000 7f081c64e640 1 -- 172.21.15.83:0/2580613895 >> v1:172.21.15.103:6800/345072327 conn(0x7f07f806df70 legacy=0x7f07f8070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:47.284 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:47.284+0000 7f081c64e640 10 monclient: shutdown 2024-03-25T19:28:47.284 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:47.284+0000 7f081c64e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:47.284 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:47.284+0000 7f081c64e640 1 -- 172.21.15.83:0/2580613895 >> v1:172.21.15.103:6789/0 conn(0x7f08140a98f0 legacy=0x7f08141d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:47.285 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:47.284+0000 7f081c64e640 1 -- 172.21.15.83:0/2580613895 shutdown_connections 2024-03-25T19:28:47.285 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:47.284+0000 7f081c64e640 1 -- 172.21.15.83:0/2580613895 >> 172.21.15.83:0/2580613895 conn(0x7f08140b7ea0 msgr2=0x7f0814074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:47.285 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:47.284+0000 7f081c64e640 1 -- 172.21.15.83:0/2580613895 shutdown_connections 2024-03-25T19:28:47.285 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:47.284+0000 7f081c64e640 1 -- 172.21.15.83:0/2580613895 wait complete. 2024-03-25T19:28:47.285 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:47.284+0000 7f081c64e640 1 librados: shutdown 2024-03-25T19:28:47.293 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:28:48.783 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:09:30.710 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:48.841 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:09:30.769 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:48.874 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:09:30.802 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:48.878 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:09:30.788 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:48.888 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:09:30.815 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:48.891 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:09:30.801 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:48.897 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:09:30.825 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:48.916 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:09:30.826 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:48.927 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:09:30.855 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:54.100 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:28:54.200 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.200+0000 7fe1885f6640 10 monclient: get_monmap_and_config 2024-03-25T19:28:54.200 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.200+0000 7fe1885f6640 10 monclient: build_initial_monmap 2024-03-25T19:28:54.200 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.200+0000 7fe1885f6640 10 monclient: monmap: 2024-03-25T19:28:54.200 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:28:54.200 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:28:54.200 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:28:54.201401+0000 2024-03-25T19:28:54.200 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:28:54.201401+0000 2024-03-25T19:28:54.200 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:54.200 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:28:54.200 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:28:54.200 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:28:54.200 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:28:54.200 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:54.201 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.201+0000 7fe1885f6640 1 Processor -- start 2024-03-25T19:28:54.201 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.201+0000 7fe1885f6640 1 -- start start 2024-03-25T19:28:54.202 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.201+0000 7fe1885f6640 10 monclient: init 2024-03-25T19:28:54.202 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.201+0000 7fe1885f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:54.202 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.201+0000 7fe1885f6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:28:54.202 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.201+0000 7fe1885f6640 10 monclient(hunting): picked mon.noname-a con 0x7fe1800b4d60 addr v1:172.21.15.83:6789/0 2024-03-25T19:28:54.202 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.201+0000 7fe1885f6640 10 monclient(hunting): picked mon.noname-c con 0x7fe1800b1730 addr v1:172.21.15.103:6789/0 2024-03-25T19:28:54.202 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.201+0000 7fe1885f6640 10 monclient(hunting): picked mon.noname-b con 0x7fe18014be90 addr v1:172.21.15.83:6790/0 2024-03-25T19:28:54.202 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.201+0000 7fe1885f6640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1800af940 con 0x7fe1800b4d60 2024-03-25T19:28:54.202 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.201+0000 7fe1885f6640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1800b0a70 con 0x7fe1800b1730 2024-03-25T19:28:54.202 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.201+0000 7fe1885f6640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1800b0d50 con 0x7fe18014be90 2024-03-25T19:28:54.202 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.201+0000 7fe1885f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:54.202 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.201+0000 7fe1885f6640 10 monclient(hunting): authenticate will time out at 6591.473676s 2024-03-25T19:28:54.204 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.204+0000 7fe185b91640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fe1800b1730 0x7fe1800b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:36674/0 (socket says 172.21.15.83:36674) 2024-03-25T19:28:54.204 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.204+0000 7fe185b91640 1 -- 172.21.15.83:0/208773494 learned_addr learned my addr 172.21.15.83:0/208773494 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:28:54.208 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.208+0000 7fe185390640 1 -- 172.21.15.83:0/208773494 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2384163733 0 0) 0x7fe1800b0a70 con 0x7fe1800b1730 2024-03-25T19:28:54.208 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.208+0000 7fe185390640 10 monclient(hunting): _negotiate 2024-03-25T19:28:54.208 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.208+0000 7fe185390640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:54.208 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.208+0000 7fe185390640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:54.209 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.208+0000 7fe185390640 10 monclient(hunting): my global_id is 4648 2024-03-25T19:28:54.209 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.208+0000 7fe185390640 1 -- 172.21.15.83:0/208773494 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe16c003520 con 0x7fe1800b1730 2024-03-25T19:28:54.216 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.216+0000 7fe185390640 1 -- 172.21.15.83:0/208773494 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2517750822 0 0) 0x7fe16c003520 con 0x7fe1800b1730 2024-03-25T19:28:54.216 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.216+0000 7fe185390640 1 -- 172.21.15.83:0/208773494 >> v1:172.21.15.83:6790/0 conn(0x7fe18014be90 legacy=0x7fe18014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:28:54.216 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.216+0000 7fe185390640 1 -- 172.21.15.83:0/208773494 >> v1:172.21.15.83:6789/0 conn(0x7fe1800b4d60 legacy=0x7fe18013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:28:54.216 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.216+0000 7fe185390640 10 monclient: _finish_hunting 0 2024-03-25T19:28:54.216 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.216+0000 7fe185390640 1 monclient: found mon.noname-c 2024-03-25T19:28:54.216 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.216+0000 7fe185390640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:54.216 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.216+0000 7fe185390640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.103:6789/0 2024-03-25T19:28:54.216 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.216+0000 7fe185390640 1 -- 172.21.15.83:0/208773494 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1800b1030 con 0x7fe1800b1730 2024-03-25T19:28:54.216 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.216+0000 7fe185390640 10 monclient: _finish_auth 0 2024-03-25T19:28:54.216 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.216+0000 7fe185390640 10 monclient: _check_auth_tickets 2024-03-25T19:28:54.216 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.216+0000 7fe185390640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:54.216 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.216+0000 7fe185390640 1 -- 172.21.15.83:0/208773494 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fe170001690 con 0x7fe1800b1730 2024-03-25T19:28:54.217 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.216+0000 7fe185390640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:54.217 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.216+0000 7fe185390640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:28:54.217 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.216+0000 7fe185390640 10 monclient: dump: 2024-03-25T19:28:54.217 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:28:54.217 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:28:54.217 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:54.217 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:54.217 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:54.217 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:28:54.217 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:28:54.217 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:28:54.217 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:28:54.217 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:54.217 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.216+0000 7fe1885f6640 5 monclient: authenticate success, global_id 4648 2024-03-25T19:28:54.217 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.216+0000 7fe1885f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:54.225 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.225+0000 7fe185390640 1 -- 172.21.15.83:0/208773494 <== mon.1 v1:172.21.15.103:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe170001860 con 0x7fe1800b1730 2024-03-25T19:28:54.225 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.225+0000 7fe185390640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:54.225 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.225+0000 7fe185390640 1 -- 172.21.15.83:0/208773494 <== mon.1 v1:172.21.15.103:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fe170001cb0 con 0x7fe1800b1730 2024-03-25T19:28:54.225 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.225+0000 7fe1885f6640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:54.225 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.225+0000 7fe1885f6640 4 set_mon_vals no callback set 2024-03-25T19:28:54.225 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.225+0000 7fe185390640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:54.225 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.225+0000 7fe185390640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:54.225 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.225+0000 7fe185390640 10 monclient: dump: 2024-03-25T19:28:54.225 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:28:54.226 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:28:54.226 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:54.226 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:54.226 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:54.226 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:28:54.226 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:28:54.226 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:28:54.226 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:28:54.226 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:54.227 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.226+0000 7fe1885f6640 10 monclient: shutdown 2024-03-25T19:28:54.227 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.226+0000 7fe1885f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:54.227 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.226+0000 7fe1885f6640 1 -- 172.21.15.83:0/208773494 >> v1:172.21.15.103:6789/0 conn(0x7fe1800b1730 legacy=0x7fe1800b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:54.228 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.226+0000 7fe1885f6640 1 -- 172.21.15.83:0/208773494 shutdown_connections 2024-03-25T19:28:54.228 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.226+0000 7fe1885f6640 1 -- 172.21.15.83:0/208773494 >> 172.21.15.83:0/208773494 conn(0x7fe1800617a0 msgr2=0x7fe180061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:54.228 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.226+0000 7fe1885f6640 1 -- 172.21.15.83:0/208773494 shutdown_connections 2024-03-25T19:28:54.228 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.226+0000 7fe1885f6640 1 -- 172.21.15.83:0/208773494 wait complete. 2024-03-25T19:28:54.228 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.227+0000 7fe1885f6640 10 monclient: build_initial_monmap 2024-03-25T19:28:54.228 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.227+0000 7fe1885f6640 10 monclient: monmap: 2024-03-25T19:28:54.228 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:28:54.228 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:54.228 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:28:54.228 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:28:54.228 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:54.228 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:28:54.228 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:28:54.228 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:28:54.228 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:28:54.228 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:54.229 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.228+0000 7fe1885f6640 1 librados: starting msgr at 2024-03-25T19:28:54.229 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.228+0000 7fe1885f6640 1 librados: starting objecter 2024-03-25T19:28:54.229 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.228+0000 7fe1885f6640 1 Processor -- start 2024-03-25T19:28:54.229 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.228+0000 7fe1885f6640 1 -- start start 2024-03-25T19:28:54.229 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.228+0000 7fe1885f6640 1 librados: setting wanted keys 2024-03-25T19:28:54.229 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.228+0000 7fe1885f6640 1 librados: calling monclient init 2024-03-25T19:28:54.229 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.228+0000 7fe1885f6640 10 monclient: init 2024-03-25T19:28:54.230 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.228+0000 7fe1885f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:54.230 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.228+0000 7fe1885f6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:28:54.230 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.228+0000 7fe1885f6640 10 monclient(hunting): picked mon.noname-c con 0x7fe1800b1730 addr v1:172.21.15.83:6790/0 2024-03-25T19:28:54.230 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.228+0000 7fe1885f6640 10 monclient(hunting): picked mon.noname-b con 0x7fe1800b4d60 addr v1:172.21.15.103:6789/0 2024-03-25T19:28:54.230 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.228+0000 7fe1885f6640 10 monclient(hunting): picked mon.noname-a con 0x7fe18014be90 addr v1:172.21.15.83:6789/0 2024-03-25T19:28:54.230 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.228+0000 7fe1885f6640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe18013a1c0 con 0x7fe18014be90 2024-03-25T19:28:54.230 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.228+0000 7fe1885f6640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe180142bf0 con 0x7fe1800b4d60 2024-03-25T19:28:54.230 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.228+0000 7fe1885f6640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe180138c40 con 0x7fe1800b1730 2024-03-25T19:28:54.230 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.228+0000 7fe1885f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:54.230 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.228+0000 7fe1885f6640 10 monclient(hunting): authenticate will time out at 6591.500638s 2024-03-25T19:28:54.231 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.231+0000 7fe186392640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fe1800b1730 0x7fe18013a8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:37850/0 (socket says 172.21.15.83:37850) 2024-03-25T19:28:54.231 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.231+0000 7fe186392640 1 -- 172.21.15.83:0/959200775 learned_addr learned my addr 172.21.15.83:0/959200775 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:28:54.231 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.232+0000 7fe176ffd640 10 client.?.objecter ms_handle_connect 0x7fe1800b1730 2024-03-25T19:28:54.231 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.232+0000 7fe176ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:54.239 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.239+0000 7fe176ffd640 1 -- 172.21.15.83:0/959200775 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1387412454 0 0) 0x7fe180138c40 con 0x7fe1800b1730 2024-03-25T19:28:54.239 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.239+0000 7fe176ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:54.239 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.239+0000 7fe176ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:54.239 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.239+0000 7fe176ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:54.239 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.239+0000 7fe176ffd640 10 monclient(hunting): my global_id is 4652 2024-03-25T19:28:54.240 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.239+0000 7fe176ffd640 1 -- 172.21.15.83:0/959200775 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe164004eb0 con 0x7fe1800b1730 2024-03-25T19:28:54.244 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.244+0000 7fe176ffd640 10 client.?.objecter ms_handle_connect 0x7fe1800b4d60 2024-03-25T19:28:54.244 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.244+0000 7fe176ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:54.249 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.249+0000 7fe176ffd640 1 -- 172.21.15.83:0/959200775 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3465483276 0 0) 0x7fe164004eb0 con 0x7fe1800b1730 2024-03-25T19:28:54.249 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.249+0000 7fe176ffd640 1 -- 172.21.15.83:0/959200775 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe180138c40 con 0x7fe1800b1730 2024-03-25T19:28:54.249 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.249+0000 7fe176ffd640 1 -- 172.21.15.83:0/959200775 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fe17c001960 con 0x7fe1800b1730 2024-03-25T19:28:54.249 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.249+0000 7fe176ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:54.250 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.249+0000 7fe176ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:28:54.250 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.249+0000 7fe176ffd640 10 monclient(hunting): dump: 2024-03-25T19:28:54.250 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:28:54.250 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:28:54.250 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:54.250 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:54.250 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:54.250 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:28:54.250 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:28:54.250 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:28:54.250 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:28:54.250 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:54.250 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.249+0000 7fe176ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:54.253 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.253+0000 7fe176ffd640 1 -- 172.21.15.83:0/959200775 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 556186841 0 0) 0x7fe180142bf0 con 0x7fe1800b4d60 2024-03-25T19:28:54.253 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.253+0000 7fe176ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:54.253 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.253+0000 7fe176ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:54.253 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.253+0000 7fe176ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:54.253 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.253+0000 7fe176ffd640 10 monclient(hunting): my global_id is 4651 2024-03-25T19:28:54.254 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.253+0000 7fe176ffd640 1 -- 172.21.15.83:0/959200775 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe164004eb0 con 0x7fe1800b4d60 2024-03-25T19:28:54.256 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.255+0000 7fe176ffd640 1 -- 172.21.15.83:0/959200775 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3635703087 0 0) 0x7fe180138c40 con 0x7fe1800b1730 2024-03-25T19:28:54.256 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.255+0000 7fe176ffd640 1 -- 172.21.15.83:0/959200775 >> v1:172.21.15.103:6789/0 conn(0x7fe1800b4d60 legacy=0x7fe1801db1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:54.256 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.255+0000 7fe176ffd640 1 -- 172.21.15.83:0/959200775 >> v1:172.21.15.83:6789/0 conn(0x7fe18014be90 legacy=0x7fe180134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:54.256 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.255+0000 7fe176ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:28:54.256 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.255+0000 7fe176ffd640 1 monclient: found mon.c 2024-03-25T19:28:54.256 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.255+0000 7fe176ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:54.256 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.255+0000 7fe176ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:28:54.256 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.255+0000 7fe176ffd640 1 -- 172.21.15.83:0/959200775 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe180138dd0 con 0x7fe1800b1730 2024-03-25T19:28:54.256 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.255+0000 7fe176ffd640 10 monclient: _finish_auth 0 2024-03-25T19:28:54.256 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.255+0000 7fe176ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:28:54.256 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.255+0000 7fe176ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:54.257 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.255+0000 7fe1885f6640 5 monclient: authenticate success, global_id 4652 2024-03-25T19:28:54.257 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.255+0000 7fe1885f6640 10 monclient: _renew_subs 2024-03-25T19:28:54.257 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.255+0000 7fe1885f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:28:54.257 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.255+0000 7fe1885f6640 1 -- 172.21.15.83:0/959200775 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1801355c0 con 0x7fe1800b1730 2024-03-25T19:28:54.257 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.255+0000 7fe1885f6640 10 client.4652.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:54.257 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.255+0000 7fe1885f6640 10 monclient: _renew_subs 2024-03-25T19:28:54.257 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.255+0000 7fe1885f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:28:54.257 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.255+0000 7fe1885f6640 1 -- 172.21.15.83:0/959200775 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe180139100 con 0x7fe1800b1730 2024-03-25T19:28:54.257 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.256+0000 7fe1885f6640 1 librados: init done 2024-03-25T19:28:54.257 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.257+0000 7fe1885f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:54.257 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.257+0000 7fe1885f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:54.257 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.257+0000 7fe1885f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:28:54.258 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.257+0000 7fe1885f6640 1 -- 172.21.15.83:0/959200775 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe180139450 con 0x7fe1800b1730 2024-03-25T19:28:54.550 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.550+0000 7fe176ffd640 1 -- 172.21.15.83:0/959200775 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe17c001c40 con 0x7fe1800b1730 2024-03-25T19:28:54.551 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.550+0000 7fe176ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:54.551 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.550+0000 7fe176ffd640 1 -- 172.21.15.83:0/959200775 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fe17c003a40 con 0x7fe1800b1730 2024-03-25T19:28:54.551 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.550+0000 7fe176ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:54.551 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.550+0000 7fe176ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:28:54.551 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.550+0000 7fe176ffd640 10 monclient: dump: 2024-03-25T19:28:54.551 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:28:54.551 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:28:54.551 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:54.551 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:28:54.551 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:54.551 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:28:54.551 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:28:54.552 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:28:54.552 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:28:54.552 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:28:54.552 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.550+0000 7fe184b8f640 4 set_mon_vals no callback set 2024-03-25T19:28:54.552 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.551+0000 7fe176ffd640 1 -- 172.21.15.83:0/959200775 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7fe17c003c60 con 0x7fe1800b1730 2024-03-25T19:28:54.553 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.552+0000 7fe185b91640 10 monclient: get_auth_request con 0x7fe16406ddc0 auth_method 0 2024-03-25T19:28:54.553 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.552+0000 7fe176ffd640 10 client.4652.objecter ms_handle_connect 0x7fe16406ddc0 2024-03-25T19:28:54.557 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.558+0000 7fe176ffd640 1 -- 172.21.15.83:0/959200775 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7fe17c054830 con 0x7fe1800b1730 2024-03-25T19:28:54.558 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.558+0000 7fe176ffd640 10 client.4652.objecter ms_dispatch 0x7fe180000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:28:54.558 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.558+0000 7fe176ffd640 3 client.4652.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:28:54.558 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.558+0000 7fe176ffd640 3 client.4652.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:28:54.558 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.558+0000 7fe176ffd640 20 client.4652.objecter dump_active .. 0 homeless 2024-03-25T19:28:54.558 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.558+0000 7fe176ffd640 1 -- 172.21.15.83:0/959200775 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe17c050340 con 0x7fe1800b1730 2024-03-25T19:28:54.558 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.558+0000 7fe176ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:54.558 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.558+0000 7fe176ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:54.792 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.792+0000 7fe1885f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:54.792 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.792+0000 7fe1885f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:54.792 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.792+0000 7fe1885f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:28:54.792 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.792+0000 7fe1885f6640 1 -- 172.21.15.83:0/959200775 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe1800627b0 con 0x7fe1800b1730 2024-03-25T19:28:54.825 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.826+0000 7fe176ffd640 1 -- 172.21.15.83:0/959200775 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7fe17c052270 con 0x7fe1800b1730 2024-03-25T19:28:54.825 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.826+0000 7fe176ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:54.826 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.826+0000 7fe176ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:28:54.826 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:28:54.826 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:28:54.828 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.828+0000 7fe1885f6640 10 librados: watch_flush enter 2024-03-25T19:28:54.829 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.828+0000 7fe1885f6640 10 librados: watch_flush exit 2024-03-25T19:28:54.829 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.828+0000 7fe1885f6640 20 client.4652.objecter shutdown clearing up homeless session... 2024-03-25T19:28:54.829 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.828+0000 7fe1885f6640 10 client.4652.objecter successfully canceled tick 2024-03-25T19:28:54.829 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.828+0000 7fe1885f6640 1 -- 172.21.15.83:0/959200775 >> v1:172.21.15.103:6800/345072327 conn(0x7fe16406ddc0 legacy=0x7fe164070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:54.829 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.828+0000 7fe1885f6640 10 monclient: shutdown 2024-03-25T19:28:54.829 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.828+0000 7fe1885f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:54.829 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.828+0000 7fe1885f6640 1 -- 172.21.15.83:0/959200775 >> v1:172.21.15.83:6790/0 conn(0x7fe1800b1730 legacy=0x7fe18013a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:54.829 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.828+0000 7fe1885f6640 1 -- 172.21.15.83:0/959200775 shutdown_connections 2024-03-25T19:28:54.829 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.829+0000 7fe1885f6640 1 -- 172.21.15.83:0/959200775 >> 172.21.15.83:0/959200775 conn(0x7fe1800617a0 msgr2=0x7fe180061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:54.829 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.829+0000 7fe1885f6640 1 -- 172.21.15.83:0/959200775 shutdown_connections 2024-03-25T19:28:54.829 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.829+0000 7fe1885f6640 1 -- 172.21.15.83:0/959200775 wait complete. 2024-03-25T19:28:54.829 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:28:54.829+0000 7fe1885f6640 1 librados: shutdown 2024-03-25T19:28:54.839 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:29:01.647 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:29:01.762 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.761+0000 7f0b1a78f640 10 monclient: get_monmap_and_config 2024-03-25T19:29:01.762 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.761+0000 7f0b1a78f640 10 monclient: build_initial_monmap 2024-03-25T19:29:01.762 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.761+0000 7f0b1a78f640 10 monclient: monmap: 2024-03-25T19:29:01.762 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:29:01.762 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:29:01.762 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:29:01.763576+0000 2024-03-25T19:29:01.762 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:29:01.763576+0000 2024-03-25T19:29:01.762 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:01.762 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:29:01.763 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:29:01.763 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:29:01.763 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:29:01.763 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:01.764 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.762+0000 7f0b1a78f640 1 Processor -- start 2024-03-25T19:29:01.764 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.762+0000 7f0b1a78f640 1 -- start start 2024-03-25T19:29:01.764 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.762+0000 7f0b1a78f640 10 monclient: init 2024-03-25T19:29:01.764 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.763+0000 7f0b1a78f640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:01.764 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.763+0000 7f0b1a78f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:29:01.764 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.763+0000 7f0b1a78f640 10 monclient(hunting): picked mon.noname-c con 0x7f0b140a98f0 addr v1:172.21.15.103:6789/0 2024-03-25T19:29:01.764 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.763+0000 7f0b1a78f640 10 monclient(hunting): picked mon.noname-b con 0x7f0b1413f710 addr v1:172.21.15.83:6790/0 2024-03-25T19:29:01.765 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.763+0000 7f0b1a78f640 10 monclient(hunting): picked mon.noname-a con 0x7f0b141430c0 addr v1:172.21.15.83:6789/0 2024-03-25T19:29:01.765 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.763+0000 7f0b1a78f640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b14142520 con 0x7f0b141430c0 2024-03-25T19:29:01.765 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.763+0000 7f0b1a78f640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b140b28a0 con 0x7f0b140a98f0 2024-03-25T19:29:01.765 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.763+0000 7f0b1a78f640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b140b2a30 con 0x7f0b1413f710 2024-03-25T19:29:01.765 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.763+0000 7f0b1a78f640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:01.765 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.763+0000 7f0b1a78f640 10 monclient(hunting): authenticate will time out at 6599.036195s 2024-03-25T19:29:01.769 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.769+0000 7f0b137fe640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f0b1413f710 0x7f0b141429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:52426/0 (socket says 172.21.15.83:52426) 2024-03-25T19:29:01.769 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.769+0000 7f0b137fe640 1 -- 172.21.15.83:0/3047206012 learned_addr learned my addr 172.21.15.83:0/3047206012 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:29:01.783 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.782+0000 7f0b12ffd640 1 -- 172.21.15.83:0/3047206012 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 924404056 0 0) 0x7f0b140b28a0 con 0x7f0b140a98f0 2024-03-25T19:29:01.783 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.782+0000 7f0b12ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:01.783 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.782+0000 7f0b12ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:01.783 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.782+0000 7f0b12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:01.783 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.782+0000 7f0b12ffd640 10 monclient(hunting): my global_id is 4654 2024-03-25T19:29:01.783 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.782+0000 7f0b12ffd640 1 -- 172.21.15.83:0/3047206012 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0afc003520 con 0x7f0b140a98f0 2024-03-25T19:29:01.786 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.785+0000 7f0b12ffd640 1 -- 172.21.15.83:0/3047206012 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3354658505 0 0) 0x7f0b140b2a30 con 0x7f0b1413f710 2024-03-25T19:29:01.786 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.785+0000 7f0b12ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:01.786 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.785+0000 7f0b12ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:01.786 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.785+0000 7f0b12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:01.786 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.785+0000 7f0b12ffd640 10 monclient(hunting): my global_id is 4655 2024-03-25T19:29:01.786 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.785+0000 7f0b12ffd640 1 -- 172.21.15.83:0/3047206012 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b140b28a0 con 0x7f0b1413f710 2024-03-25T19:29:01.797 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.796+0000 7f0b12ffd640 1 -- 172.21.15.83:0/3047206012 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1165880060 0 0) 0x7f0b14142520 con 0x7f0b141430c0 2024-03-25T19:29:01.797 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.796+0000 7f0b12ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:01.797 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.796+0000 7f0b12ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:01.797 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.796+0000 7f0b12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:01.797 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.796+0000 7f0b12ffd640 10 monclient(hunting): my global_id is 4638 2024-03-25T19:29:01.797 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.796+0000 7f0b12ffd640 1 -- 172.21.15.83:0/3047206012 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b140b2a30 con 0x7f0b141430c0 2024-03-25T19:29:01.807 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.806+0000 7f0b12ffd640 1 -- 172.21.15.83:0/3047206012 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 469108302 0 0) 0x7f0b140b28a0 con 0x7f0b1413f710 2024-03-25T19:29:01.807 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.806+0000 7f0b12ffd640 1 -- 172.21.15.83:0/3047206012 >> v1:172.21.15.103:6789/0 conn(0x7f0b140a98f0 legacy=0x7f0b140a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:01.807 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.806+0000 7f0b12ffd640 1 -- 172.21.15.83:0/3047206012 >> v1:172.21.15.83:6789/0 conn(0x7f0b141430c0 legacy=0x7f0b1414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:01.807 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.806+0000 7f0b12ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:29:01.807 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.806+0000 7f0b12ffd640 1 monclient: found mon.noname-b 2024-03-25T19:29:01.807 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.806+0000 7f0b12ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:01.807 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.806+0000 7f0b12ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.83:6790/0 2024-03-25T19:29:01.808 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.806+0000 7f0b12ffd640 1 -- 172.21.15.83:0/3047206012 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b140b44c0 con 0x7f0b1413f710 2024-03-25T19:29:01.808 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.806+0000 7f0b12ffd640 10 monclient: _finish_auth 0 2024-03-25T19:29:01.808 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.806+0000 7f0b12ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:29:01.808 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.806+0000 7f0b12ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:01.808 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.806+0000 7f0b12ffd640 1 -- 172.21.15.83:0/3047206012 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f0b08001670 con 0x7f0b1413f710 2024-03-25T19:29:01.808 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.806+0000 7f0b12ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:01.808 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.806+0000 7f0b12ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:29:01.808 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.806+0000 7f0b12ffd640 10 monclient: dump: 2024-03-25T19:29:01.808 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:29:01.808 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:29:01.808 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:01.808 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:01.808 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:01.808 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:29:01.808 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:29:01.808 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:29:01.809 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:29:01.809 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:01.809 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.806+0000 7f0b1a78f640 5 monclient: authenticate success, global_id 4655 2024-03-25T19:29:01.809 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.806+0000 7f0b1a78f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:01.818 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.817+0000 7f0b12ffd640 1 -- 172.21.15.83:0/3047206012 <== mon.2 v1:172.21.15.83:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b080017d0 con 0x7f0b1413f710 2024-03-25T19:29:01.818 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.817+0000 7f0b12ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:01.818 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.817+0000 7f0b12ffd640 1 -- 172.21.15.83:0/3047206012 <== mon.2 v1:172.21.15.83:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f0b08001be0 con 0x7f0b1413f710 2024-03-25T19:29:01.818 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.817+0000 7f0b12ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:01.818 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.817+0000 7f0b12ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:29:01.818 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.817+0000 7f0b12ffd640 10 monclient: dump: 2024-03-25T19:29:01.818 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:29:01.818 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:29:01.818 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:01.818 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:01.818 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:01.819 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:29:01.819 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:29:01.819 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:29:01.819 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:29:01.819 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:01.819 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.817+0000 7f0b1a78f640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:01.819 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.817+0000 7f0b1a78f640 4 set_mon_vals no callback set 2024-03-25T19:29:01.820 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.818+0000 7f0b1a78f640 10 monclient: shutdown 2024-03-25T19:29:01.820 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.818+0000 7f0b1a78f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:01.820 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.818+0000 7f0b1a78f640 1 -- 172.21.15.83:0/3047206012 >> v1:172.21.15.83:6790/0 conn(0x7f0b1413f710 legacy=0x7f0b141429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:01.820 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.818+0000 7f0b1a78f640 1 -- 172.21.15.83:0/3047206012 shutdown_connections 2024-03-25T19:29:01.820 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.818+0000 7f0b1a78f640 1 -- 172.21.15.83:0/3047206012 >> 172.21.15.83:0/3047206012 conn(0x7f0b140b7ea0 msgr2=0x7f0b140b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:01.820 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.818+0000 7f0b1a78f640 1 -- 172.21.15.83:0/3047206012 shutdown_connections 2024-03-25T19:29:01.820 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.818+0000 7f0b1a78f640 1 -- 172.21.15.83:0/3047206012 wait complete. 2024-03-25T19:29:01.820 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.819+0000 7f0b1a78f640 10 monclient: build_initial_monmap 2024-03-25T19:29:01.820 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.819+0000 7f0b1a78f640 10 monclient: monmap: 2024-03-25T19:29:01.820 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:29:01.820 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:01.821 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:29:01.821 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:29:01.821 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:01.821 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:29:01.821 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:29:01.821 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:29:01.821 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:29:01.821 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:01.822 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.820+0000 7f0b1a78f640 1 librados: starting msgr at 2024-03-25T19:29:01.822 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.820+0000 7f0b1a78f640 1 librados: starting objecter 2024-03-25T19:29:01.822 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.820+0000 7f0b1a78f640 1 Processor -- start 2024-03-25T19:29:01.822 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.820+0000 7f0b1a78f640 1 -- start start 2024-03-25T19:29:01.822 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.820+0000 7f0b1a78f640 1 librados: setting wanted keys 2024-03-25T19:29:01.822 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.820+0000 7f0b1a78f640 1 librados: calling monclient init 2024-03-25T19:29:01.822 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.820+0000 7f0b1a78f640 10 monclient: init 2024-03-25T19:29:01.822 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.821+0000 7f0b1a78f640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:01.822 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.821+0000 7f0b1a78f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:29:01.822 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.821+0000 7f0b1a78f640 10 monclient(hunting): picked mon.noname-a con 0x7f0b140a98f0 addr v1:172.21.15.83:6789/0 2024-03-25T19:29:01.822 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.821+0000 7f0b1a78f640 10 monclient(hunting): picked mon.noname-c con 0x7f0b1413f710 addr v1:172.21.15.83:6790/0 2024-03-25T19:29:01.822 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.821+0000 7f0b1a78f640 10 monclient(hunting): picked mon.noname-b con 0x7f0b141430c0 addr v1:172.21.15.103:6789/0 2024-03-25T19:29:01.823 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.821+0000 7f0b1a78f640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b14138ed0 con 0x7f0b140a98f0 2024-03-25T19:29:01.823 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.821+0000 7f0b1a78f640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b140abc80 con 0x7f0b141430c0 2024-03-25T19:29:01.823 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.821+0000 7f0b1a78f640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b140abf60 con 0x7f0b1413f710 2024-03-25T19:29:01.823 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.821+0000 7f0b1a78f640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:01.823 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.821+0000 7f0b1a78f640 10 monclient(hunting): authenticate will time out at 6599.094219s 2024-03-25T19:29:01.830 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.830+0000 7f0b137fe640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f0b1413f710 0x7f0b140ac660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:52442/0 (socket says 172.21.15.83:52442) 2024-03-25T19:29:01.830 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.830+0000 7f0b137fe640 1 -- 172.21.15.83:0/2582703498 learned_addr learned my addr 172.21.15.83:0/2582703498 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:29:01.831 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.830+0000 7f0b10ff9640 10 client.?.objecter ms_handle_connect 0x7f0b1413f710 2024-03-25T19:29:01.831 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.831+0000 7f0b10ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:01.837 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.837+0000 7f0b10ff9640 10 client.?.objecter ms_handle_connect 0x7f0b140a98f0 2024-03-25T19:29:01.837 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.837+0000 7f0b10ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:01.841 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.841+0000 7f0b10ff9640 1 -- 172.21.15.83:0/2582703498 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1674791248 0 0) 0x7f0b140abf60 con 0x7f0b1413f710 2024-03-25T19:29:01.842 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.841+0000 7f0b10ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:29:01.842 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.841+0000 7f0b10ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:01.842 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.841+0000 7f0b10ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:01.842 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.841+0000 7f0b10ff9640 10 monclient(hunting): my global_id is 4658 2024-03-25T19:29:01.842 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.841+0000 7f0b10ff9640 1 -- 172.21.15.83:0/2582703498 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b00004eb0 con 0x7f0b1413f710 2024-03-25T19:29:01.845 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.844+0000 7f0b10ff9640 10 client.?.objecter ms_handle_connect 0x7f0b141430c0 2024-03-25T19:29:01.845 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.844+0000 7f0b10ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:01.854 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.853+0000 7f0b10ff9640 1 -- 172.21.15.83:0/2582703498 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 886377371 0 0) 0x7f0b14138ed0 con 0x7f0b140a98f0 2024-03-25T19:29:01.854 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.853+0000 7f0b10ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:29:01.854 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.853+0000 7f0b10ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:01.854 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.853+0000 7f0b10ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:01.854 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.853+0000 7f0b10ff9640 10 monclient(hunting): my global_id is 4641 2024-03-25T19:29:01.854 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.853+0000 7f0b10ff9640 1 -- 172.21.15.83:0/2582703498 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b140abf60 con 0x7f0b140a98f0 2024-03-25T19:29:01.855 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.854+0000 7f0b10ff9640 1 -- 172.21.15.83:0/2582703498 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3936972410 0 0) 0x7f0b140abc80 con 0x7f0b141430c0 2024-03-25T19:29:01.855 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.854+0000 7f0b10ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:29:01.855 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.854+0000 7f0b10ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:01.855 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.854+0000 7f0b10ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:01.855 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.854+0000 7f0b10ff9640 10 monclient(hunting): my global_id is 4657 2024-03-25T19:29:01.855 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.854+0000 7f0b10ff9640 1 -- 172.21.15.83:0/2582703498 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b14138ed0 con 0x7f0b141430c0 2024-03-25T19:29:01.856 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.855+0000 7f0b10ff9640 1 -- 172.21.15.83:0/2582703498 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2865033023 0 0) 0x7f0b00004eb0 con 0x7f0b1413f710 2024-03-25T19:29:01.856 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.855+0000 7f0b10ff9640 1 -- 172.21.15.83:0/2582703498 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b140abc80 con 0x7f0b1413f710 2024-03-25T19:29:01.856 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.855+0000 7f0b10ff9640 1 -- 172.21.15.83:0/2582703498 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f0b08001c20 con 0x7f0b1413f710 2024-03-25T19:29:01.856 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.855+0000 7f0b10ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:01.856 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.855+0000 7f0b10ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:29:01.856 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.855+0000 7f0b10ff9640 10 monclient(hunting): dump: 2024-03-25T19:29:01.856 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:29:01.856 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:29:01.857 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:01.857 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:01.857 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:01.857 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:29:01.857 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:29:01.857 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:29:01.857 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:29:01.857 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:01.857 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.855+0000 7f0b10ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:01.866 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.864+0000 7f0b10ff9640 1 -- 172.21.15.83:0/2582703498 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 247529304 0 0) 0x7f0b140abc80 con 0x7f0b1413f710 2024-03-25T19:29:01.866 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.864+0000 7f0b10ff9640 1 -- 172.21.15.83:0/2582703498 >> v1:172.21.15.103:6789/0 conn(0x7f0b141430c0 legacy=0x7f0b141e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:01.866 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.864+0000 7f0b10ff9640 1 -- 172.21.15.83:0/2582703498 >> v1:172.21.15.83:6789/0 conn(0x7f0b140a98f0 legacy=0x7f0b141387c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:01.867 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.864+0000 7f0b10ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:29:01.867 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.864+0000 7f0b10ff9640 1 monclient: found mon.c 2024-03-25T19:29:01.867 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.864+0000 7f0b10ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:01.867 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.864+0000 7f0b10ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:29:01.867 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.864+0000 7f0b10ff9640 1 -- 172.21.15.83:0/2582703498 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b14139170 con 0x7f0b1413f710 2024-03-25T19:29:01.867 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.864+0000 7f0b10ff9640 10 monclient: _finish_auth 0 2024-03-25T19:29:01.867 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.864+0000 7f0b10ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:29:01.867 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.864+0000 7f0b10ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:01.867 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.864+0000 7f0b1a78f640 5 monclient: authenticate success, global_id 4658 2024-03-25T19:29:01.867 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.864+0000 7f0b1a78f640 10 monclient: _renew_subs 2024-03-25T19:29:01.867 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.864+0000 7f0b1a78f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:29:01.867 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.864+0000 7f0b1a78f640 1 -- 172.21.15.83:0/2582703498 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b140acd70 con 0x7f0b1413f710 2024-03-25T19:29:01.867 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.864+0000 7f0b1a78f640 10 client.4658.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:01.867 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.864+0000 7f0b1a78f640 10 monclient: _renew_subs 2024-03-25T19:29:01.867 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.864+0000 7f0b1a78f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:29:01.868 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.864+0000 7f0b1a78f640 1 -- 172.21.15.83:0/2582703498 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b14136c30 con 0x7f0b1413f710 2024-03-25T19:29:01.868 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.864+0000 7f0b1a78f640 1 librados: init done 2024-03-25T19:29:01.868 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.865+0000 7f0b1a78f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:01.868 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.865+0000 7f0b1a78f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:01.868 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.865+0000 7f0b1a78f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:29:01.868 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.865+0000 7f0b1a78f640 1 -- 172.21.15.83:0/2582703498 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b14136690 con 0x7f0b1413f710 2024-03-25T19:29:01.872 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.871+0000 7f0b10ff9640 1 -- 172.21.15.83:0/2582703498 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b08004a90 con 0x7f0b1413f710 2024-03-25T19:29:01.872 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.871+0000 7f0b10ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:01.872 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:01.871+0000 7f0b127fc640 4 set_mon_vals no callback set 2024-03-25T19:29:02.163 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:02.163+0000 7f0b10ff9640 1 -- 172.21.15.83:0/2582703498 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f0b08004de0 con 0x7f0b1413f710 2024-03-25T19:29:02.164 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:02.163+0000 7f0b10ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:02.164 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:02.163+0000 7f0b10ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:29:02.164 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:02.163+0000 7f0b10ff9640 10 monclient: dump: 2024-03-25T19:29:02.164 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:29:02.164 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:29:02.164 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:02.164 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:02.164 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:02.164 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:29:02.164 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:29:02.164 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:29:02.164 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:29:02.164 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:02.165 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:02.163+0000 7f0b10ff9640 1 -- 172.21.15.83:0/2582703498 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7f0b08004fc0 con 0x7f0b1413f710 2024-03-25T19:29:02.165 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:02.164+0000 7f0b13fff640 10 monclient: get_auth_request con 0x7f0b0006dd50 auth_method 0 2024-03-25T19:29:02.165 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:02.164+0000 7f0b10ff9640 10 client.4658.objecter ms_handle_connect 0x7f0b0006dd50 2024-03-25T19:29:02.173 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:02.172+0000 7f0b10ff9640 1 -- 172.21.15.83:0/2582703498 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7f0b080549f0 con 0x7f0b1413f710 2024-03-25T19:29:02.173 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:02.172+0000 7f0b10ff9640 10 client.4658.objecter ms_dispatch 0x7f0b14000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:29:02.173 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:02.172+0000 7f0b10ff9640 3 client.4658.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:29:02.173 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:02.172+0000 7f0b10ff9640 3 client.4658.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:29:02.173 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:02.172+0000 7f0b10ff9640 20 client.4658.objecter dump_active .. 0 homeless 2024-03-25T19:29:02.173 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:02.172+0000 7f0b10ff9640 1 -- 172.21.15.83:0/2582703498 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0b08050330 con 0x7f0b1413f710 2024-03-25T19:29:02.174 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:02.172+0000 7f0b10ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:02.174 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:02.172+0000 7f0b10ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:02.389 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:02.388+0000 7f0b1a78f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:02.389 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:02.388+0000 7f0b1a78f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:02.389 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:02.388+0000 7f0b1a78f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:29:02.390 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:02.388+0000 7f0b1a78f640 1 -- 172.21.15.83:0/2582703498 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b140627b0 con 0x7f0b1413f710 2024-03-25T19:29:02.458 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:02.457+0000 7f0b10ff9640 1 -- 172.21.15.83:0/2582703498 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7f0b08052430 con 0x7f0b1413f710 2024-03-25T19:29:02.458 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:02.458+0000 7f0b10ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:02.458 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:02.458+0000 7f0b10ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:02.459 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:29:02.459 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:29:02.461 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:02.460+0000 7f0b1a78f640 10 librados: watch_flush enter 2024-03-25T19:29:02.461 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:02.460+0000 7f0b1a78f640 10 librados: watch_flush exit 2024-03-25T19:29:02.461 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:02.460+0000 7f0b1a78f640 20 client.4658.objecter shutdown clearing up homeless session... 2024-03-25T19:29:02.461 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:02.460+0000 7f0b1a78f640 10 client.4658.objecter successfully canceled tick 2024-03-25T19:29:02.461 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:02.460+0000 7f0b1a78f640 1 -- 172.21.15.83:0/2582703498 >> v1:172.21.15.103:6800/345072327 conn(0x7f0b0006dd50 legacy=0x7f0b00070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:02.462 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:02.460+0000 7f0b1a78f640 10 monclient: shutdown 2024-03-25T19:29:02.462 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:02.460+0000 7f0b1a78f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:02.462 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:02.460+0000 7f0b1a78f640 1 -- 172.21.15.83:0/2582703498 >> v1:172.21.15.83:6790/0 conn(0x7f0b1413f710 legacy=0x7f0b140ac660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:02.462 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:02.460+0000 7f0b1a78f640 1 -- 172.21.15.83:0/2582703498 shutdown_connections 2024-03-25T19:29:02.462 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:02.460+0000 7f0b1a78f640 1 -- 172.21.15.83:0/2582703498 >> 172.21.15.83:0/2582703498 conn(0x7f0b140b7ea0 msgr2=0x7f0b14074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:02.462 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:02.460+0000 7f0b1a78f640 1 -- 172.21.15.83:0/2582703498 shutdown_connections 2024-03-25T19:29:02.462 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:02.460+0000 7f0b1a78f640 1 -- 172.21.15.83:0/2582703498 wait complete. 2024-03-25T19:29:02.462 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:02.460+0000 7f0b1a78f640 1 librados: shutdown 2024-03-25T19:29:02.473 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:29:08.880 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:09:50.807 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:08.888 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:09:50.815 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:08.928 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:09:50.856 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:08.942 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:09:50.869 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:08.944 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:09:50.854 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:08.945 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:09:50.873 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:08.948 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:09:50.858 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:08.970 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:09:50.880 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:08.982 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:09:50.909 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:09.279 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:29:09.392 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.392+0000 7fa29aa43640 10 monclient: get_monmap_and_config 2024-03-25T19:29:09.392 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.392+0000 7fa29aa43640 10 monclient: build_initial_monmap 2024-03-25T19:29:09.392 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.392+0000 7fa29aa43640 10 monclient: monmap: 2024-03-25T19:29:09.392 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:29:09.392 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:29:09.392 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:29:09.393319+0000 2024-03-25T19:29:09.392 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:29:09.393319+0000 2024-03-25T19:29:09.392 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:09.392 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:29:09.392 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:29:09.393 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:29:09.393 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:29:09.393 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:09.394 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.393+0000 7fa29aa43640 1 Processor -- start 2024-03-25T19:29:09.394 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.393+0000 7fa29aa43640 1 -- start start 2024-03-25T19:29:09.394 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.393+0000 7fa29aa43640 10 monclient: init 2024-03-25T19:29:09.394 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.393+0000 7fa29aa43640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:09.394 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.393+0000 7fa29aa43640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:29:09.394 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.393+0000 7fa29aa43640 10 monclient(hunting): picked mon.noname-c con 0x7fa2940a98f0 addr v1:172.21.15.103:6789/0 2024-03-25T19:29:09.394 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.393+0000 7fa29aa43640 10 monclient(hunting): picked mon.noname-b con 0x7fa29413f710 addr v1:172.21.15.83:6790/0 2024-03-25T19:29:09.394 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.393+0000 7fa29aa43640 10 monclient(hunting): picked mon.noname-a con 0x7fa2941430c0 addr v1:172.21.15.83:6789/0 2024-03-25T19:29:09.394 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.393+0000 7fa29aa43640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa294142520 con 0x7fa2941430c0 2024-03-25T19:29:09.395 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.393+0000 7fa29aa43640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2940b28a0 con 0x7fa2940a98f0 2024-03-25T19:29:09.395 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.393+0000 7fa29aa43640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2940b2a30 con 0x7fa29413f710 2024-03-25T19:29:09.395 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.393+0000 7fa29aa43640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:09.395 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.393+0000 7fa29aa43640 10 monclient(hunting): authenticate will time out at 6606.665585s 2024-03-25T19:29:09.402 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.402+0000 7fa298fe0640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fa2941430c0 0x7fa29414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:53476/0 (socket says 172.21.15.83:53476) 2024-03-25T19:29:09.402 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.402+0000 7fa298fe0640 1 -- 172.21.15.83:0/722521839 learned_addr learned my addr 172.21.15.83:0/722521839 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:29:09.425 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.424+0000 7fa292ffd640 1 -- 172.21.15.83:0/722521839 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 850175511 0 0) 0x7fa2940b28a0 con 0x7fa2940a98f0 2024-03-25T19:29:09.425 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.424+0000 7fa292ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:09.425 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.424+0000 7fa292ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:09.425 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.424+0000 7fa292ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:09.425 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.424+0000 7fa292ffd640 10 monclient(hunting): my global_id is 4660 2024-03-25T19:29:09.425 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.424+0000 7fa292ffd640 1 -- 172.21.15.83:0/722521839 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa27c003520 con 0x7fa2940a98f0 2024-03-25T19:29:09.425 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.424+0000 7fa292ffd640 1 -- 172.21.15.83:0/722521839 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1929641129 0 0) 0x7fa2940b2a30 con 0x7fa29413f710 2024-03-25T19:29:09.425 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.424+0000 7fa292ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:09.425 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.424+0000 7fa292ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:09.425 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.424+0000 7fa292ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:09.425 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.424+0000 7fa292ffd640 10 monclient(hunting): my global_id is 4661 2024-03-25T19:29:09.425 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.424+0000 7fa292ffd640 1 -- 172.21.15.83:0/722521839 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2940b28a0 con 0x7fa29413f710 2024-03-25T19:29:09.425 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.425+0000 7fa292ffd640 1 -- 172.21.15.83:0/722521839 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2958181280 0 0) 0x7fa294142520 con 0x7fa2941430c0 2024-03-25T19:29:09.426 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.425+0000 7fa292ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:09.426 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.425+0000 7fa292ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:09.426 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.425+0000 7fa292ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:09.426 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.425+0000 7fa292ffd640 10 monclient(hunting): my global_id is 4644 2024-03-25T19:29:09.426 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.425+0000 7fa292ffd640 1 -- 172.21.15.83:0/722521839 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2940b2a30 con 0x7fa2941430c0 2024-03-25T19:29:09.435 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.435+0000 7fa292ffd640 1 -- 172.21.15.83:0/722521839 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 148245517 0 0) 0x7fa2940b2a30 con 0x7fa2941430c0 2024-03-25T19:29:09.435 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.435+0000 7fa292ffd640 1 -- 172.21.15.83:0/722521839 >> v1:172.21.15.83:6790/0 conn(0x7fa29413f710 legacy=0x7fa2941429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:09.435 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.435+0000 7fa292ffd640 1 -- 172.21.15.83:0/722521839 >> v1:172.21.15.103:6789/0 conn(0x7fa2940a98f0 legacy=0x7fa2940a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:09.435 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.435+0000 7fa292ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:29:09.435 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.435+0000 7fa292ffd640 1 monclient: found mon.noname-a 2024-03-25T19:29:09.435 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.435+0000 7fa292ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:09.435 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.435+0000 7fa292ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.83:6789/0 2024-03-25T19:29:09.435 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.435+0000 7fa292ffd640 1 -- 172.21.15.83:0/722521839 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2940b44c0 con 0x7fa2941430c0 2024-03-25T19:29:09.435 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.435+0000 7fa292ffd640 10 monclient: _finish_auth 0 2024-03-25T19:29:09.435 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.435+0000 7fa292ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:29:09.436 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.435+0000 7fa292ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:09.436 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.435+0000 7fa292ffd640 1 -- 172.21.15.83:0/722521839 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fa28c001930 con 0x7fa2941430c0 2024-03-25T19:29:09.436 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.435+0000 7fa29aa43640 5 monclient: authenticate success, global_id 4644 2024-03-25T19:29:09.436 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.435+0000 7fa29aa43640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:09.436 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.435+0000 7fa292ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:09.436 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.435+0000 7fa292ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:29:09.436 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.435+0000 7fa292ffd640 10 monclient: dump: 2024-03-25T19:29:09.436 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:29:09.436 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:29:09.436 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:09.436 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:09.436 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:09.436 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:29:09.436 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:29:09.436 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:29:09.436 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:29:09.437 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:09.437 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.435+0000 7fa29aa43640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:09.441 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.442+0000 7fa292ffd640 1 -- 172.21.15.83:0/722521839 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa28c001a90 con 0x7fa2941430c0 2024-03-25T19:29:09.441 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.442+0000 7fa292ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:09.442 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.442+0000 7fa292ffd640 1 -- 172.21.15.83:0/722521839 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fa28c001ee0 con 0x7fa2941430c0 2024-03-25T19:29:09.442 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.442+0000 7fa29aa43640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:09.442 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.442+0000 7fa29aa43640 4 set_mon_vals no callback set 2024-03-25T19:29:09.442 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.442+0000 7fa292ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:09.442 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.442+0000 7fa292ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:29:09.442 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.442+0000 7fa292ffd640 10 monclient: dump: 2024-03-25T19:29:09.442 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:29:09.442 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:29:09.442 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:09.442 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:09.442 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:09.442 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:29:09.442 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:29:09.442 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:29:09.443 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:29:09.443 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:09.444 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.442+0000 7fa29aa43640 10 monclient: shutdown 2024-03-25T19:29:09.444 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.442+0000 7fa29aa43640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:09.444 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.442+0000 7fa29aa43640 1 -- 172.21.15.83:0/722521839 >> v1:172.21.15.83:6789/0 conn(0x7fa2941430c0 legacy=0x7fa29414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:09.444 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.443+0000 7fa29aa43640 1 -- 172.21.15.83:0/722521839 shutdown_connections 2024-03-25T19:29:09.444 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.443+0000 7fa29aa43640 1 -- 172.21.15.83:0/722521839 >> 172.21.15.83:0/722521839 conn(0x7fa2940b7ea0 msgr2=0x7fa2940b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:09.444 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.443+0000 7fa29aa43640 1 -- 172.21.15.83:0/722521839 shutdown_connections 2024-03-25T19:29:09.444 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.443+0000 7fa29aa43640 1 -- 172.21.15.83:0/722521839 wait complete. 2024-03-25T19:29:09.444 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.444+0000 7fa29aa43640 10 monclient: build_initial_monmap 2024-03-25T19:29:09.444 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.444+0000 7fa29aa43640 10 monclient: monmap: 2024-03-25T19:29:09.444 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:29:09.445 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:09.445 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:29:09.445 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:29:09.445 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:09.445 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:29:09.445 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:29:09.445 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:29:09.445 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:29:09.445 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:09.445 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.444+0000 7fa29aa43640 1 librados: starting msgr at 2024-03-25T19:29:09.445 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.444+0000 7fa29aa43640 1 librados: starting objecter 2024-03-25T19:29:09.445 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.444+0000 7fa29aa43640 1 Processor -- start 2024-03-25T19:29:09.445 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.444+0000 7fa29aa43640 1 -- start start 2024-03-25T19:29:09.446 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.444+0000 7fa29aa43640 1 librados: setting wanted keys 2024-03-25T19:29:09.446 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.444+0000 7fa29aa43640 1 librados: calling monclient init 2024-03-25T19:29:09.446 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.444+0000 7fa29aa43640 10 monclient: init 2024-03-25T19:29:09.447 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.445+0000 7fa29aa43640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:09.447 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.445+0000 7fa29aa43640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:29:09.447 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.445+0000 7fa29aa43640 10 monclient(hunting): picked mon.noname-c con 0x7fa2940a98f0 addr v1:172.21.15.83:6790/0 2024-03-25T19:29:09.447 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.445+0000 7fa29aa43640 10 monclient(hunting): picked mon.noname-a con 0x7fa29413f710 addr v1:172.21.15.83:6789/0 2024-03-25T19:29:09.447 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.445+0000 7fa29aa43640 10 monclient(hunting): picked mon.noname-b con 0x7fa2941430c0 addr v1:172.21.15.103:6789/0 2024-03-25T19:29:09.447 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.445+0000 7fa29aa43640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa29406f800 con 0x7fa29413f710 2024-03-25T19:29:09.447 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.445+0000 7fa29aa43640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2940706e0 con 0x7fa2941430c0 2024-03-25T19:29:09.447 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.445+0000 7fa29aa43640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2940709c0 con 0x7fa2940a98f0 2024-03-25T19:29:09.447 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.445+0000 7fa29aa43640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:09.447 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.445+0000 7fa29aa43640 10 monclient(hunting): authenticate will time out at 6606.717570s 2024-03-25T19:29:09.452 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.453+0000 7fa293fff640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fa2940a98f0 0x7fa29406ed20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:35740/0 (socket says 172.21.15.83:35740) 2024-03-25T19:29:09.453 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.453+0000 7fa293fff640 1 -- 172.21.15.83:0/370475967 learned_addr learned my addr 172.21.15.83:0/370475967 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:29:09.454 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.455+0000 7fa290ff9640 10 client.?.objecter ms_handle_connect 0x7fa2940a98f0 2024-03-25T19:29:09.454 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.455+0000 7fa290ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:09.456 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.456+0000 7fa290ff9640 10 client.?.objecter ms_handle_connect 0x7fa2941430c0 2024-03-25T19:29:09.456 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.456+0000 7fa290ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:09.459 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.459+0000 7fa290ff9640 10 client.?.objecter ms_handle_connect 0x7fa29413f710 2024-03-25T19:29:09.459 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.459+0000 7fa290ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:09.462 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.463+0000 7fa290ff9640 1 -- 172.21.15.83:0/370475967 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2773722397 0 0) 0x7fa2940709c0 con 0x7fa2940a98f0 2024-03-25T19:29:09.462 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.463+0000 7fa290ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:29:09.463 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.463+0000 7fa290ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:09.463 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.463+0000 7fa290ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:09.463 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.463+0000 7fa290ff9640 10 monclient(hunting): my global_id is 4664 2024-03-25T19:29:09.463 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.463+0000 7fa290ff9640 1 -- 172.21.15.83:0/370475967 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa280004eb0 con 0x7fa2940a98f0 2024-03-25T19:29:09.481 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.482+0000 7fa290ff9640 1 -- 172.21.15.83:0/370475967 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4064589209 0 0) 0x7fa280004eb0 con 0x7fa2940a98f0 2024-03-25T19:29:09.482 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.482+0000 7fa290ff9640 1 -- 172.21.15.83:0/370475967 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2940709c0 con 0x7fa2940a98f0 2024-03-25T19:29:09.482 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.482+0000 7fa290ff9640 1 -- 172.21.15.83:0/370475967 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fa2840019b0 con 0x7fa2940a98f0 2024-03-25T19:29:09.482 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.482+0000 7fa290ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:09.482 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.482+0000 7fa290ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:29:09.482 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.482+0000 7fa290ff9640 10 monclient(hunting): dump: 2024-03-25T19:29:09.482 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:29:09.482 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:29:09.482 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:09.482 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:09.482 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:09.482 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:29:09.482 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:29:09.482 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:29:09.482 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:29:09.483 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:09.483 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.482+0000 7fa290ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:09.483 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.482+0000 7fa290ff9640 1 -- 172.21.15.83:0/370475967 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1696678937 0 0) 0x7fa29406f800 con 0x7fa29413f710 2024-03-25T19:29:09.483 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.482+0000 7fa290ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:29:09.483 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.482+0000 7fa290ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:09.483 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.482+0000 7fa290ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:09.483 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.482+0000 7fa290ff9640 10 monclient(hunting): my global_id is 4647 2024-03-25T19:29:09.483 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.482+0000 7fa290ff9640 1 -- 172.21.15.83:0/370475967 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa280004eb0 con 0x7fa29413f710 2024-03-25T19:29:09.485 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.485+0000 7fa290ff9640 1 -- 172.21.15.83:0/370475967 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 918370679 0 0) 0x7fa2940706e0 con 0x7fa2941430c0 2024-03-25T19:29:09.485 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.485+0000 7fa290ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:29:09.485 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.485+0000 7fa290ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:09.485 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.485+0000 7fa290ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:09.485 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.485+0000 7fa290ff9640 10 monclient(hunting): my global_id is 4663 2024-03-25T19:29:09.485 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.485+0000 7fa290ff9640 1 -- 172.21.15.83:0/370475967 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa29406f800 con 0x7fa2941430c0 2024-03-25T19:29:09.497 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.496+0000 7fa290ff9640 1 -- 172.21.15.83:0/370475967 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2490710331 0 0) 0x7fa2940709c0 con 0x7fa2940a98f0 2024-03-25T19:29:09.497 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.496+0000 7fa290ff9640 1 -- 172.21.15.83:0/370475967 >> v1:172.21.15.103:6789/0 conn(0x7fa2941430c0 legacy=0x7fa2941e6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:09.497 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.496+0000 7fa290ff9640 1 -- 172.21.15.83:0/370475967 >> v1:172.21.15.83:6789/0 conn(0x7fa29413f710 legacy=0x7fa2941df5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:09.497 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.496+0000 7fa290ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:29:09.497 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.496+0000 7fa290ff9640 1 monclient: found mon.c 2024-03-25T19:29:09.497 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.496+0000 7fa290ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:09.497 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.496+0000 7fa290ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:29:09.497 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.496+0000 7fa290ff9640 1 -- 172.21.15.83:0/370475967 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa294070de0 con 0x7fa2940a98f0 2024-03-25T19:29:09.497 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.496+0000 7fa290ff9640 10 monclient: _finish_auth 0 2024-03-25T19:29:09.497 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.496+0000 7fa290ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:29:09.497 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.497+0000 7fa290ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:09.498 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.497+0000 7fa29aa43640 5 monclient: authenticate success, global_id 4664 2024-03-25T19:29:09.498 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.497+0000 7fa29aa43640 10 monclient: _renew_subs 2024-03-25T19:29:09.498 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.497+0000 7fa29aa43640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:29:09.498 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.497+0000 7fa29aa43640 1 -- 172.21.15.83:0/370475967 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa29406f430 con 0x7fa2940a98f0 2024-03-25T19:29:09.498 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.497+0000 7fa29aa43640 10 client.4664.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:09.498 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.497+0000 7fa29aa43640 10 monclient: _renew_subs 2024-03-25T19:29:09.498 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.497+0000 7fa29aa43640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:29:09.498 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.497+0000 7fa29aa43640 1 -- 172.21.15.83:0/370475967 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2941dfee0 con 0x7fa2940a98f0 2024-03-25T19:29:09.498 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.497+0000 7fa29aa43640 1 librados: init done 2024-03-25T19:29:09.499 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.498+0000 7fa29aa43640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:09.499 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.498+0000 7fa29aa43640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:09.499 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.498+0000 7fa29aa43640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:29:09.499 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.498+0000 7fa29aa43640 1 -- 172.21.15.83:0/370475967 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa29414f2d0 con 0x7fa2940a98f0 2024-03-25T19:29:09.801 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.801+0000 7fa290ff9640 1 -- 172.21.15.83:0/370475967 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa284001c50 con 0x7fa2940a98f0 2024-03-25T19:29:09.801 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.801+0000 7fa290ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:09.801 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.801+0000 7fa290ff9640 1 -- 172.21.15.83:0/370475967 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fa284003a60 con 0x7fa2940a98f0 2024-03-25T19:29:09.801 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.801+0000 7fa290ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:09.801 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.801+0000 7fa290ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:29:09.801 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.801+0000 7fa290ff9640 10 monclient: dump: 2024-03-25T19:29:09.801 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:29:09.801 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:29:09.802 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:09.802 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:09.802 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:09.802 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:29:09.802 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:29:09.802 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:29:09.802 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:29:09.802 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:09.802 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.801+0000 7fa2927fc640 4 set_mon_vals no callback set 2024-03-25T19:29:09.803 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.802+0000 7fa290ff9640 1 -- 172.21.15.83:0/370475967 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7fa284003c80 con 0x7fa2940a98f0 2024-03-25T19:29:09.803 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.803+0000 7fa2937fe640 10 monclient: get_auth_request con 0x7fa28006de80 auth_method 0 2024-03-25T19:29:09.803 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.803+0000 7fa290ff9640 10 client.4664.objecter ms_handle_connect 0x7fa28006de80 2024-03-25T19:29:09.808 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.808+0000 7fa290ff9640 1 -- 172.21.15.83:0/370475967 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7fa2840547c0 con 0x7fa2940a98f0 2024-03-25T19:29:09.808 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.808+0000 7fa290ff9640 10 client.4664.objecter ms_dispatch 0x7fa294000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:29:09.808 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.808+0000 7fa290ff9640 3 client.4664.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:29:09.808 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.808+0000 7fa290ff9640 3 client.4664.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:29:09.808 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.808+0000 7fa290ff9640 20 client.4664.objecter dump_active .. 0 homeless 2024-03-25T19:29:09.808 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.808+0000 7fa290ff9640 1 -- 172.21.15.83:0/370475967 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa2840502d0 con 0x7fa2940a98f0 2024-03-25T19:29:09.808 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.808+0000 7fa290ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:09.808 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:09.808+0000 7fa290ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:10.026 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:10.026+0000 7fa29aa43640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:10.026 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:10.026+0000 7fa29aa43640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:10.027 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:10.026+0000 7fa29aa43640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:29:10.027 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:10.026+0000 7fa29aa43640 1 -- 172.21.15.83:0/370475967 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa294070f70 con 0x7fa2940a98f0 2024-03-25T19:29:10.053 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:10.053+0000 7fa290ff9640 1 -- 172.21.15.83:0/370475967 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7fa284052200 con 0x7fa2940a98f0 2024-03-25T19:29:10.053 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:10.053+0000 7fa290ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:10.053 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:10.053+0000 7fa290ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:10.053 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:29:10.053 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:29:10.056 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:10.056+0000 7fa29aa43640 10 librados: watch_flush enter 2024-03-25T19:29:10.056 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:10.056+0000 7fa29aa43640 10 librados: watch_flush exit 2024-03-25T19:29:10.056 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:10.056+0000 7fa29aa43640 20 client.4664.objecter shutdown clearing up homeless session... 2024-03-25T19:29:10.056 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:10.056+0000 7fa29aa43640 10 client.4664.objecter successfully canceled tick 2024-03-25T19:29:10.056 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:10.056+0000 7fa29aa43640 1 -- 172.21.15.83:0/370475967 >> v1:172.21.15.103:6800/345072327 conn(0x7fa28006de80 legacy=0x7fa280070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:10.056 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:10.056+0000 7fa29aa43640 10 monclient: shutdown 2024-03-25T19:29:10.056 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:10.056+0000 7fa29aa43640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:10.056 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:10.056+0000 7fa29aa43640 1 -- 172.21.15.83:0/370475967 >> v1:172.21.15.83:6790/0 conn(0x7fa2940a98f0 legacy=0x7fa29406ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:10.056 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:10.056+0000 7fa29aa43640 1 -- 172.21.15.83:0/370475967 shutdown_connections 2024-03-25T19:29:10.057 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:10.056+0000 7fa29aa43640 1 -- 172.21.15.83:0/370475967 >> 172.21.15.83:0/370475967 conn(0x7fa2940b7ea0 msgr2=0x7fa294074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:10.057 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:10.056+0000 7fa29aa43640 1 -- 172.21.15.83:0/370475967 shutdown_connections 2024-03-25T19:29:10.057 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:10.056+0000 7fa29aa43640 1 -- 172.21.15.83:0/370475967 wait complete. 2024-03-25T19:29:10.057 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:10.056+0000 7fa29aa43640 1 librados: shutdown 2024-03-25T19:29:10.068 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:29:16.875 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:29:16.992 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:16.992+0000 7f4039250640 10 monclient: get_monmap_and_config 2024-03-25T19:29:16.992 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:16.992+0000 7f4039250640 10 monclient: build_initial_monmap 2024-03-25T19:29:16.992 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:16.992+0000 7f4039250640 10 monclient: monmap: 2024-03-25T19:29:16.992 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:29:16.992 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:29:16.992 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:29:16.993426+0000 2024-03-25T19:29:16.992 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:29:16.993426+0000 2024-03-25T19:29:16.992 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:16.993 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:29:16.993 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:29:16.993 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:29:16.993 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:29:16.993 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:16.993 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:16.993+0000 7f4039250640 1 Processor -- start 2024-03-25T19:29:16.993 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:16.993+0000 7f4039250640 1 -- start start 2024-03-25T19:29:16.993 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:16.993+0000 7f4039250640 10 monclient: init 2024-03-25T19:29:16.994 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:16.994+0000 7f4039250640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:16.994 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:16.994+0000 7f4039250640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:29:16.994 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:16.994+0000 7f4039250640 10 monclient(hunting): picked mon.noname-b con 0x7f40340a98f0 addr v1:172.21.15.83:6790/0 2024-03-25T19:29:16.994 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:16.994+0000 7f4039250640 10 monclient(hunting): picked mon.noname-c con 0x7f403413f710 addr v1:172.21.15.103:6789/0 2024-03-25T19:29:16.994 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:16.994+0000 7f4039250640 10 monclient(hunting): picked mon.noname-a con 0x7f40341430c0 addr v1:172.21.15.83:6789/0 2024-03-25T19:29:16.994 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:16.994+0000 7f4039250640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4034142520 con 0x7f40341430c0 2024-03-25T19:29:16.994 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:16.994+0000 7f4039250640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40340b28a0 con 0x7f403413f710 2024-03-25T19:29:16.994 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:16.994+0000 7f4039250640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40340b2a30 con 0x7f40340a98f0 2024-03-25T19:29:16.994 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:16.994+0000 7f4039250640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:16.995 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:16.994+0000 7f4039250640 10 monclient(hunting): authenticate will time out at 6614.266348s 2024-03-25T19:29:17.003 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.003+0000 7f403259c640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f403413f710 0x7f40341429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:55694/0 (socket says 172.21.15.83:55694) 2024-03-25T19:29:17.003 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.003+0000 7f403259c640 1 -- 172.21.15.83:0/377303264 learned_addr learned my addr 172.21.15.83:0/377303264 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:29:17.024 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.024+0000 7f4031d9b640 1 -- 172.21.15.83:0/377303264 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1787415306 0 0) 0x7f40340b2a30 con 0x7f40340a98f0 2024-03-25T19:29:17.024 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.025+0000 7f4031d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:29:17.024 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.025+0000 7f4031d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:17.024 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.025+0000 7f4031d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:17.024 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.025+0000 7f4031d9b640 10 monclient(hunting): my global_id is 4667 2024-03-25T19:29:17.024 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.025+0000 7f4031d9b640 1 -- 172.21.15.83:0/377303264 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f401c003520 con 0x7f40340a98f0 2024-03-25T19:29:17.027 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.027+0000 7f4031d9b640 1 -- 172.21.15.83:0/377303264 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3184258357 0 0) 0x7f4034142520 con 0x7f40341430c0 2024-03-25T19:29:17.027 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.028+0000 7f4031d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:29:17.027 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.028+0000 7f4031d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:17.027 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.028+0000 7f4031d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:17.027 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.028+0000 7f4031d9b640 10 monclient(hunting): my global_id is 4650 2024-03-25T19:29:17.028 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.028+0000 7f4031d9b640 1 -- 172.21.15.83:0/377303264 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40340b2a30 con 0x7f40341430c0 2024-03-25T19:29:17.028 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.029+0000 7f4031d9b640 1 -- 172.21.15.83:0/377303264 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4205718530 0 0) 0x7f40340b28a0 con 0x7f403413f710 2024-03-25T19:29:17.028 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.029+0000 7f4031d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:29:17.029 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.029+0000 7f4031d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:17.029 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.029+0000 7f4031d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:17.029 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.029+0000 7f4031d9b640 10 monclient(hunting): my global_id is 4666 2024-03-25T19:29:17.029 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.029+0000 7f4031d9b640 1 -- 172.21.15.83:0/377303264 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4034142520 con 0x7f403413f710 2024-03-25T19:29:17.048 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.048+0000 7f4031d9b640 1 -- 172.21.15.83:0/377303264 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1994448641 0 0) 0x7f4034142520 con 0x7f403413f710 2024-03-25T19:29:17.048 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.048+0000 7f4031d9b640 1 -- 172.21.15.83:0/377303264 >> v1:172.21.15.83:6790/0 conn(0x7f40340a98f0 legacy=0x7f40340a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:17.049 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.048+0000 7f4031d9b640 1 -- 172.21.15.83:0/377303264 >> v1:172.21.15.83:6789/0 conn(0x7f40341430c0 legacy=0x7f403414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:17.049 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.048+0000 7f4031d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:29:17.049 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.048+0000 7f4031d9b640 1 monclient: found mon.noname-c 2024-03-25T19:29:17.049 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.048+0000 7f4031d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:17.049 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.048+0000 7f4031d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.103:6789/0 2024-03-25T19:29:17.049 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.048+0000 7f4031d9b640 1 -- 172.21.15.83:0/377303264 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40340b44c0 con 0x7f403413f710 2024-03-25T19:29:17.049 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.048+0000 7f4031d9b640 10 monclient: _finish_auth 0 2024-03-25T19:29:17.049 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.048+0000 7f4031d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:29:17.049 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.048+0000 7f4031d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:17.049 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.048+0000 7f4039250640 5 monclient: authenticate success, global_id 4666 2024-03-25T19:29:17.049 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.048+0000 7f4031d9b640 1 -- 172.21.15.83:0/377303264 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f40240015b0 con 0x7f403413f710 2024-03-25T19:29:17.049 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.048+0000 7f4031d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:17.049 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.048+0000 7f4031d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:29:17.049 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.048+0000 7f4031d9b640 10 monclient: dump: 2024-03-25T19:29:17.050 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:29:17.050 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:29:17.050 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:17.050 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:17.050 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:17.050 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:29:17.050 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:29:17.050 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:29:17.050 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:29:17.050 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:17.050 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.048+0000 7f4039250640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:17.079 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.080+0000 7f4031d9b640 1 -- 172.21.15.83:0/377303264 <== mon.1 v1:172.21.15.103:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4024001710 con 0x7f403413f710 2024-03-25T19:29:17.080 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.080+0000 7f4031d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:17.080 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.080+0000 7f4031d9b640 1 -- 172.21.15.83:0/377303264 <== mon.1 v1:172.21.15.103:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f4024001b60 con 0x7f403413f710 2024-03-25T19:29:17.080 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.080+0000 7f4039250640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:17.080 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.080+0000 7f4039250640 4 set_mon_vals no callback set 2024-03-25T19:29:17.080 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.080+0000 7f4031d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:17.080 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.080+0000 7f4031d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:17.080 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.080+0000 7f4031d9b640 10 monclient: dump: 2024-03-25T19:29:17.080 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:29:17.080 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:29:17.080 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:17.080 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:17.080 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:17.081 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:29:17.081 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:29:17.081 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:29:17.081 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:29:17.081 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:17.083 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.080+0000 7f4039250640 10 monclient: shutdown 2024-03-25T19:29:17.083 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.081+0000 7f4039250640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:17.083 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.081+0000 7f4039250640 1 -- 172.21.15.83:0/377303264 >> v1:172.21.15.103:6789/0 conn(0x7f403413f710 legacy=0x7f40341429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:17.083 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.081+0000 7f4039250640 1 -- 172.21.15.83:0/377303264 shutdown_connections 2024-03-25T19:29:17.083 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.081+0000 7f4039250640 1 -- 172.21.15.83:0/377303264 >> 172.21.15.83:0/377303264 conn(0x7f40340b7ea0 msgr2=0x7f40340b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:17.083 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.081+0000 7f4039250640 1 -- 172.21.15.83:0/377303264 shutdown_connections 2024-03-25T19:29:17.083 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.081+0000 7f4039250640 1 -- 172.21.15.83:0/377303264 wait complete. 2024-03-25T19:29:17.083 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.082+0000 7f4039250640 10 monclient: build_initial_monmap 2024-03-25T19:29:17.084 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.082+0000 7f4039250640 10 monclient: monmap: 2024-03-25T19:29:17.084 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:29:17.084 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:17.084 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:29:17.084 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:29:17.084 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:17.084 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:29:17.084 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:29:17.084 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:29:17.084 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:29:17.084 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:17.084 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.082+0000 7f4039250640 1 librados: starting msgr at 2024-03-25T19:29:17.084 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.082+0000 7f4039250640 1 librados: starting objecter 2024-03-25T19:29:17.084 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.082+0000 7f4039250640 1 Processor -- start 2024-03-25T19:29:17.084 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.082+0000 7f4039250640 1 -- start start 2024-03-25T19:29:17.085 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.082+0000 7f4039250640 1 librados: setting wanted keys 2024-03-25T19:29:17.085 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.082+0000 7f4039250640 1 librados: calling monclient init 2024-03-25T19:29:17.085 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.082+0000 7f4039250640 10 monclient: init 2024-03-25T19:29:17.085 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.083+0000 7f4039250640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:17.085 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.083+0000 7f4039250640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:29:17.085 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.083+0000 7f4039250640 10 monclient(hunting): picked mon.noname-b con 0x7f40340a98f0 addr v1:172.21.15.103:6789/0 2024-03-25T19:29:17.085 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.083+0000 7f4039250640 10 monclient(hunting): picked mon.noname-a con 0x7f403413f710 addr v1:172.21.15.83:6789/0 2024-03-25T19:29:17.085 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.083+0000 7f4039250640 10 monclient(hunting): picked mon.noname-c con 0x7f40341430c0 addr v1:172.21.15.83:6790/0 2024-03-25T19:29:17.085 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.083+0000 7f4039250640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40340abb90 con 0x7f403413f710 2024-03-25T19:29:17.085 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.083+0000 7f4039250640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40341da920 con 0x7f40340a98f0 2024-03-25T19:29:17.085 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.083+0000 7f4039250640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4034137790 con 0x7f40341430c0 2024-03-25T19:29:17.085 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.083+0000 7f4039250640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:17.085 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.083+0000 7f4039250640 10 monclient(hunting): authenticate will time out at 6614.355102s 2024-03-25T19:29:17.086 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.084+0000 7f4032d9d640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f40340a98f0 0x7f40341dae40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:55760/0 (socket says 172.21.15.83:55760) 2024-03-25T19:29:17.086 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.084+0000 7f4032d9d640 1 -- 172.21.15.83:0/2586232433 learned_addr learned my addr 172.21.15.83:0/2586232433 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:29:17.086 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.086+0000 7f401b7fe640 10 client.?.objecter ms_handle_connect 0x7f40340a98f0 2024-03-25T19:29:17.086 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.086+0000 7f401b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:17.088 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.089+0000 7f401b7fe640 10 client.?.objecter ms_handle_connect 0x7f40341430c0 2024-03-25T19:29:17.088 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.089+0000 7f401b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:17.146 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.146+0000 7f401b7fe640 10 client.?.objecter ms_handle_connect 0x7f403413f710 2024-03-25T19:29:17.146 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.146+0000 7f401b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:17.156 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.156+0000 7f401b7fe640 1 -- 172.21.15.83:0/2586232433 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2175314693 0 0) 0x7f40340abb90 con 0x7f403413f710 2024-03-25T19:29:17.156 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.156+0000 7f401b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:29:17.156 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.156+0000 7f401b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:17.156 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.156+0000 7f401b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:17.156 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.156+0000 7f401b7fe640 10 monclient(hunting): my global_id is 4653 2024-03-25T19:29:17.156 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.156+0000 7f401b7fe640 1 -- 172.21.15.83:0/2586232433 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4020004eb0 con 0x7f403413f710 2024-03-25T19:29:17.176 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.176+0000 7f401b7fe640 1 -- 172.21.15.83:0/2586232433 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4013556534 0 0) 0x7f4020004eb0 con 0x7f403413f710 2024-03-25T19:29:17.176 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.177+0000 7f401b7fe640 1 -- 172.21.15.83:0/2586232433 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40340abb90 con 0x7f403413f710 2024-03-25T19:29:17.177 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.177+0000 7f401b7fe640 1 -- 172.21.15.83:0/2586232433 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f4024001a20 con 0x7f403413f710 2024-03-25T19:29:17.177 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.177+0000 7f401b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:17.177 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.177+0000 7f401b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:29:17.177 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.177+0000 7f401b7fe640 10 monclient(hunting): dump: 2024-03-25T19:29:17.177 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:29:17.177 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:29:17.177 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:17.177 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:17.177 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:17.177 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:29:17.177 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:29:17.177 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:29:17.177 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:29:17.177 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:17.177 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.177+0000 7f401b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:17.193 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.193+0000 7f401b7fe640 1 -- 172.21.15.83:0/2586232433 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 574765130 0 0) 0x7f40340abb90 con 0x7f403413f710 2024-03-25T19:29:17.193 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.193+0000 7f401b7fe640 1 -- 172.21.15.83:0/2586232433 >> v1:172.21.15.83:6790/0 conn(0x7f40341430c0 legacy=0x7f40341e1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:17.194 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.193+0000 7f401b7fe640 1 -- 172.21.15.83:0/2586232433 >> v1:172.21.15.103:6789/0 conn(0x7f40340a98f0 legacy=0x7f40341dae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:17.194 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.193+0000 7f401b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:29:17.194 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.193+0000 7f401b7fe640 1 monclient: found mon.a 2024-03-25T19:29:17.194 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.193+0000 7f401b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:17.194 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.193+0000 7f401b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:29:17.194 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.193+0000 7f401b7fe640 1 -- 172.21.15.83:0/2586232433 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4034137920 con 0x7f403413f710 2024-03-25T19:29:17.194 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.193+0000 7f401b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:29:17.194 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.193+0000 7f401b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:29:17.194 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.193+0000 7f401b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:17.194 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.193+0000 7f4039250640 5 monclient: authenticate success, global_id 4653 2024-03-25T19:29:17.194 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.193+0000 7f4039250640 10 monclient: _renew_subs 2024-03-25T19:29:17.194 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.193+0000 7f4039250640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:29:17.194 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.193+0000 7f4039250640 1 -- 172.21.15.83:0/2586232433 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40341db550 con 0x7f403413f710 2024-03-25T19:29:17.195 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.193+0000 7f4039250640 10 client.4653.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:17.195 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.193+0000 7f4039250640 10 monclient: _renew_subs 2024-03-25T19:29:17.195 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.193+0000 7f4039250640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:29:17.195 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.193+0000 7f4039250640 1 -- 172.21.15.83:0/2586232433 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4034138c00 con 0x7f403413f710 2024-03-25T19:29:17.195 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.193+0000 7f4039250640 1 librados: init done 2024-03-25T19:29:17.195 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.194+0000 7f4039250640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:17.195 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.194+0000 7f4039250640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:17.196 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.194+0000 7f4039250640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:29:17.196 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.194+0000 7f4039250640 1 -- 172.21.15.83:0/2586232433 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40341dab60 con 0x7f403413f710 2024-03-25T19:29:17.498 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.498+0000 7f401b7fe640 1 -- 172.21.15.83:0/2586232433 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4024004a90 con 0x7f403413f710 2024-03-25T19:29:17.499 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.498+0000 7f401b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:17.499 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.498+0000 7f401b7fe640 1 -- 172.21.15.83:0/2586232433 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f4024004c00 con 0x7f403413f710 2024-03-25T19:29:17.499 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.498+0000 7f401b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:17.499 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.498+0000 7f401b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:29:17.499 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.498+0000 7f401b7fe640 10 monclient: dump: 2024-03-25T19:29:17.499 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:29:17.499 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:29:17.499 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:17.499 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:17.499 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:17.499 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:29:17.499 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:29:17.499 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:29:17.499 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:29:17.500 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:17.500 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.498+0000 7f403159a640 4 set_mon_vals no callback set 2024-03-25T19:29:17.500 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.499+0000 7f401b7fe640 1 -- 172.21.15.83:0/2586232433 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7f402401abd0 con 0x7f403413f710 2024-03-25T19:29:17.500 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.500+0000 7f4032d9d640 10 monclient: get_auth_request con 0x7f402006dc90 auth_method 0 2024-03-25T19:29:17.500 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.500+0000 7f401b7fe640 10 client.4653.objecter ms_handle_connect 0x7f402006dc90 2024-03-25T19:29:17.506 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.506+0000 7f401b7fe640 1 -- 172.21.15.83:0/2586232433 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7f4024054710 con 0x7f403413f710 2024-03-25T19:29:17.506 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.506+0000 7f401b7fe640 10 client.4653.objecter ms_dispatch 0x7f4034000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:29:17.506 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.506+0000 7f401b7fe640 3 client.4653.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:29:17.506 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.506+0000 7f401b7fe640 3 client.4653.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:29:17.506 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.506+0000 7f401b7fe640 20 client.4653.objecter dump_active .. 0 homeless 2024-03-25T19:29:17.506 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.506+0000 7f401b7fe640 1 -- 172.21.15.83:0/2586232433 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4024050220 con 0x7f403413f710 2024-03-25T19:29:17.506 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.506+0000 7f401b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:17.506 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.506+0000 7f401b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:17.755 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.755+0000 7f4039250640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:17.756 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.756+0000 7f4039250640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:17.756 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.756+0000 7f4039250640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:29:17.756 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.756+0000 7f4039250640 1 -- 172.21.15.83:0/2586232433 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f40340627b0 con 0x7f403413f710 2024-03-25T19:29:17.794 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.794+0000 7f401b7fe640 1 -- 172.21.15.83:0/2586232433 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7f4024052150 con 0x7f403413f710 2024-03-25T19:29:17.794 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.794+0000 7f401b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:17.794 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.794+0000 7f401b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:17.794 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:29:17.794 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:29:17.796 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.796+0000 7f4039250640 10 librados: watch_flush enter 2024-03-25T19:29:17.797 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.796+0000 7f4039250640 10 librados: watch_flush exit 2024-03-25T19:29:17.797 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.796+0000 7f4039250640 20 client.4653.objecter shutdown clearing up homeless session... 2024-03-25T19:29:17.797 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.796+0000 7f4039250640 10 client.4653.objecter successfully canceled tick 2024-03-25T19:29:17.797 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.796+0000 7f4039250640 1 -- 172.21.15.83:0/2586232433 >> v1:172.21.15.103:6800/345072327 conn(0x7f402006dc90 legacy=0x7f4020070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:17.797 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.796+0000 7f4039250640 10 monclient: shutdown 2024-03-25T19:29:17.797 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.796+0000 7f4039250640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:17.797 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.796+0000 7f4039250640 1 -- 172.21.15.83:0/2586232433 >> v1:172.21.15.83:6789/0 conn(0x7f403413f710 legacy=0x7f4034138310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:17.797 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.797+0000 7f4039250640 1 -- 172.21.15.83:0/2586232433 shutdown_connections 2024-03-25T19:29:17.797 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.797+0000 7f4039250640 1 -- 172.21.15.83:0/2586232433 >> 172.21.15.83:0/2586232433 conn(0x7f40340b7ea0 msgr2=0x7f4034074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:17.797 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.797+0000 7f4039250640 1 -- 172.21.15.83:0/2586232433 shutdown_connections 2024-03-25T19:29:17.797 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.797+0000 7f4039250640 1 -- 172.21.15.83:0/2586232433 wait complete. 2024-03-25T19:29:17.797 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:17.797+0000 7f4039250640 1 librados: shutdown 2024-03-25T19:29:17.808 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:29:24.615 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:29:24.727 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.726+0000 7fb774bc2640 10 monclient: get_monmap_and_config 2024-03-25T19:29:24.727 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.726+0000 7fb774bc2640 10 monclient: build_initial_monmap 2024-03-25T19:29:24.727 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.726+0000 7fb774bc2640 10 monclient: monmap: 2024-03-25T19:29:24.727 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:29:24.727 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:29:24.727 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:29:24.727657+0000 2024-03-25T19:29:24.727 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:29:24.727657+0000 2024-03-25T19:29:24.727 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:24.727 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:29:24.727 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:29:24.727 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:29:24.727 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:29:24.727 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:24.727 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.727+0000 7fb774bc2640 1 Processor -- start 2024-03-25T19:29:24.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.727+0000 7fb774bc2640 1 -- start start 2024-03-25T19:29:24.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.727+0000 7fb774bc2640 10 monclient: init 2024-03-25T19:29:24.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.728+0000 7fb774bc2640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:24.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.728+0000 7fb774bc2640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:29:24.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.728+0000 7fb774bc2640 10 monclient(hunting): picked mon.noname-c con 0x7fb7700a98f0 addr v1:172.21.15.103:6789/0 2024-03-25T19:29:24.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.728+0000 7fb774bc2640 10 monclient(hunting): picked mon.noname-b con 0x7fb77013f710 addr v1:172.21.15.83:6790/0 2024-03-25T19:29:24.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.728+0000 7fb774bc2640 10 monclient(hunting): picked mon.noname-a con 0x7fb7701430c0 addr v1:172.21.15.83:6789/0 2024-03-25T19:29:24.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.728+0000 7fb774bc2640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb770142520 con 0x7fb7701430c0 2024-03-25T19:29:24.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.728+0000 7fb774bc2640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7700b28a0 con 0x7fb7700a98f0 2024-03-25T19:29:24.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.728+0000 7fb774bc2640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7700b2a30 con 0x7fb77013f710 2024-03-25T19:29:24.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.728+0000 7fb774bc2640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:24.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.728+0000 7fb774bc2640 10 monclient(hunting): authenticate will time out at 6622.000353s 2024-03-25T19:29:24.738 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.738+0000 7fb76dd9b640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fb77013f710 0x7fb7701429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:50282/0 (socket says 172.21.15.83:50282) 2024-03-25T19:29:24.738 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.738+0000 7fb76dd9b640 1 -- 172.21.15.83:0/928418884 learned_addr learned my addr 172.21.15.83:0/928418884 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:29:24.753 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.752+0000 7fb76d59a640 1 -- 172.21.15.83:0/928418884 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1266312699 0 0) 0x7fb7700b2a30 con 0x7fb77013f710 2024-03-25T19:29:24.753 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.752+0000 7fb76d59a640 10 monclient(hunting): _negotiate 2024-03-25T19:29:24.753 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.752+0000 7fb76d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:24.753 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.752+0000 7fb76d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:24.753 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.752+0000 7fb76d59a640 10 monclient(hunting): my global_id is 4673 2024-03-25T19:29:24.753 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.752+0000 7fb76d59a640 1 -- 172.21.15.83:0/928418884 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb758003520 con 0x7fb77013f710 2024-03-25T19:29:24.754 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.754+0000 7fb76d59a640 1 -- 172.21.15.83:0/928418884 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4275064876 0 0) 0x7fb770142520 con 0x7fb7701430c0 2024-03-25T19:29:24.754 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.754+0000 7fb76d59a640 10 monclient(hunting): _negotiate 2024-03-25T19:29:24.754 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.754+0000 7fb76d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:24.754 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.754+0000 7fb76d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:24.754 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.755+0000 7fb76d59a640 10 monclient(hunting): my global_id is 4656 2024-03-25T19:29:24.754 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.755+0000 7fb76d59a640 1 -- 172.21.15.83:0/928418884 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7700b2a30 con 0x7fb7701430c0 2024-03-25T19:29:24.763 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.763+0000 7fb76d59a640 1 -- 172.21.15.83:0/928418884 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1082102638 0 0) 0x7fb7700b28a0 con 0x7fb7700a98f0 2024-03-25T19:29:24.763 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.763+0000 7fb76d59a640 10 monclient(hunting): _negotiate 2024-03-25T19:29:24.763 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.763+0000 7fb76d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:24.763 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.763+0000 7fb76d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:24.763 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.763+0000 7fb76d59a640 10 monclient(hunting): my global_id is 4672 2024-03-25T19:29:24.763 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.763+0000 7fb76d59a640 1 -- 172.21.15.83:0/928418884 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb770142520 con 0x7fb7700a98f0 2024-03-25T19:29:24.775 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.775+0000 7fb76d59a640 1 -- 172.21.15.83:0/928418884 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 612446220 0 0) 0x7fb758003520 con 0x7fb77013f710 2024-03-25T19:29:24.775 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.775+0000 7fb76d59a640 1 -- 172.21.15.83:0/928418884 >> v1:172.21.15.103:6789/0 conn(0x7fb7700a98f0 legacy=0x7fb7700a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:24.775 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.775+0000 7fb76d59a640 1 -- 172.21.15.83:0/928418884 >> v1:172.21.15.83:6789/0 conn(0x7fb7701430c0 legacy=0x7fb77014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:24.775 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.775+0000 7fb76d59a640 10 monclient: _finish_hunting 0 2024-03-25T19:29:24.775 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.775+0000 7fb76d59a640 1 monclient: found mon.noname-b 2024-03-25T19:29:24.775 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.775+0000 7fb76d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:24.775 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.775+0000 7fb76d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.83:6790/0 2024-03-25T19:29:24.775 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.775+0000 7fb76d59a640 1 -- 172.21.15.83:0/928418884 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7700b44c0 con 0x7fb77013f710 2024-03-25T19:29:24.776 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.775+0000 7fb76d59a640 10 monclient: _finish_auth 0 2024-03-25T19:29:24.776 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.775+0000 7fb76d59a640 10 monclient: _check_auth_tickets 2024-03-25T19:29:24.776 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.775+0000 7fb76d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:24.776 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.775+0000 7fb76d59a640 1 -- 172.21.15.83:0/928418884 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fb764001690 con 0x7fb77013f710 2024-03-25T19:29:24.776 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.775+0000 7fb774bc2640 5 monclient: authenticate success, global_id 4673 2024-03-25T19:29:24.776 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.775+0000 7fb774bc2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:24.776 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.775+0000 7fb76d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:24.776 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.775+0000 7fb76d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:29:24.776 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.775+0000 7fb76d59a640 10 monclient: dump: 2024-03-25T19:29:24.776 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:29:24.776 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:29:24.776 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:24.776 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:24.776 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:24.776 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:29:24.776 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:29:24.776 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:29:24.776 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:29:24.777 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:24.777 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.775+0000 7fb774bc2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:24.782 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.782+0000 7fb76d59a640 1 -- 172.21.15.83:0/928418884 <== mon.2 v1:172.21.15.83:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb7640017f0 con 0x7fb77013f710 2024-03-25T19:29:24.782 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.782+0000 7fb76d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:24.783 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.782+0000 7fb76d59a640 1 -- 172.21.15.83:0/928418884 <== mon.2 v1:172.21.15.83:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fb764001c00 con 0x7fb77013f710 2024-03-25T19:29:24.783 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.783+0000 7fb774bc2640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:24.783 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.783+0000 7fb774bc2640 4 set_mon_vals no callback set 2024-03-25T19:29:24.783 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.783+0000 7fb76d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:24.783 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.783+0000 7fb76d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:29:24.783 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.783+0000 7fb76d59a640 10 monclient: dump: 2024-03-25T19:29:24.783 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:29:24.783 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:29:24.783 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:24.783 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:24.783 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:24.783 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:29:24.783 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:29:24.783 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:29:24.783 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:29:24.783 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:24.784 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.783+0000 7fb774bc2640 10 monclient: shutdown 2024-03-25T19:29:24.785 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.783+0000 7fb774bc2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:24.785 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.783+0000 7fb774bc2640 1 -- 172.21.15.83:0/928418884 >> v1:172.21.15.83:6790/0 conn(0x7fb77013f710 legacy=0x7fb7701429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:24.785 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.784+0000 7fb774bc2640 1 -- 172.21.15.83:0/928418884 shutdown_connections 2024-03-25T19:29:24.785 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.784+0000 7fb774bc2640 1 -- 172.21.15.83:0/928418884 >> 172.21.15.83:0/928418884 conn(0x7fb7700b7ea0 msgr2=0x7fb7700b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:24.785 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.784+0000 7fb774bc2640 1 -- 172.21.15.83:0/928418884 shutdown_connections 2024-03-25T19:29:24.785 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.784+0000 7fb774bc2640 1 -- 172.21.15.83:0/928418884 wait complete. 2024-03-25T19:29:24.785 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.785+0000 7fb774bc2640 10 monclient: build_initial_monmap 2024-03-25T19:29:24.785 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.785+0000 7fb774bc2640 10 monclient: monmap: 2024-03-25T19:29:24.785 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:29:24.785 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:24.785 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:29:24.785 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:29:24.785 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:24.785 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:29:24.785 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:29:24.785 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:29:24.786 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:29:24.786 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:24.786 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.785+0000 7fb774bc2640 1 librados: starting msgr at 2024-03-25T19:29:24.786 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.785+0000 7fb774bc2640 1 librados: starting objecter 2024-03-25T19:29:24.786 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.785+0000 7fb774bc2640 1 Processor -- start 2024-03-25T19:29:24.786 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.785+0000 7fb774bc2640 1 -- start start 2024-03-25T19:29:24.786 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.785+0000 7fb774bc2640 1 librados: setting wanted keys 2024-03-25T19:29:24.786 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.785+0000 7fb774bc2640 1 librados: calling monclient init 2024-03-25T19:29:24.786 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.785+0000 7fb774bc2640 10 monclient: init 2024-03-25T19:29:24.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.786+0000 7fb774bc2640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:24.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.786+0000 7fb774bc2640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:29:24.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.786+0000 7fb774bc2640 10 monclient(hunting): picked mon.noname-c con 0x7fb7700a98f0 addr v1:172.21.15.83:6790/0 2024-03-25T19:29:24.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.786+0000 7fb774bc2640 10 monclient(hunting): picked mon.noname-a con 0x7fb77013f710 addr v1:172.21.15.83:6789/0 2024-03-25T19:29:24.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.786+0000 7fb774bc2640 10 monclient(hunting): picked mon.noname-b con 0x7fb7701430c0 addr v1:172.21.15.103:6789/0 2024-03-25T19:29:24.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.786+0000 7fb774bc2640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb77013be70 con 0x7fb77013f710 2024-03-25T19:29:24.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.786+0000 7fb774bc2640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb770131660 con 0x7fb7701430c0 2024-03-25T19:29:24.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.786+0000 7fb774bc2640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7701317f0 con 0x7fb7700a98f0 2024-03-25T19:29:24.788 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.786+0000 7fb774bc2640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:24.788 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.786+0000 7fb774bc2640 10 monclient(hunting): authenticate will time out at 6622.058249s 2024-03-25T19:29:24.799 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.799+0000 7fb76e59c640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fb7700a98f0 0x7fb770130f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:50332/0 (socket says 172.21.15.83:50332) 2024-03-25T19:29:24.799 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.799+0000 7fb76e59c640 1 -- 172.21.15.83:0/951638589 learned_addr learned my addr 172.21.15.83:0/951638589 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:29:24.800 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.800+0000 7fb75effd640 10 client.?.objecter ms_handle_connect 0x7fb7700a98f0 2024-03-25T19:29:24.800 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.800+0000 7fb75effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:24.801 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.802+0000 7fb75effd640 10 client.?.objecter ms_handle_connect 0x7fb77013f710 2024-03-25T19:29:24.801 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.802+0000 7fb75effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:24.803 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.804+0000 7fb75effd640 10 client.?.objecter ms_handle_connect 0x7fb7701430c0 2024-03-25T19:29:24.804 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.804+0000 7fb75effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:24.804 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.805+0000 7fb75effd640 1 -- 172.21.15.83:0/951638589 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3913533836 0 0) 0x7fb77013be70 con 0x7fb77013f710 2024-03-25T19:29:24.804 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.805+0000 7fb75effd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:24.804 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.805+0000 7fb75effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:24.805 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.805+0000 7fb75effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:24.805 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.805+0000 7fb75effd640 10 monclient(hunting): my global_id is 4659 2024-03-25T19:29:24.805 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.805+0000 7fb75effd640 1 -- 172.21.15.83:0/951638589 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb750004eb0 con 0x7fb77013f710 2024-03-25T19:29:24.811 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.812+0000 7fb75effd640 1 -- 172.21.15.83:0/951638589 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 130914795 0 0) 0x7fb7701317f0 con 0x7fb7700a98f0 2024-03-25T19:29:24.811 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.812+0000 7fb75effd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:24.811 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.812+0000 7fb75effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:24.812 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.812+0000 7fb75effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:24.812 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.812+0000 7fb75effd640 10 monclient(hunting): my global_id is 4676 2024-03-25T19:29:24.812 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.812+0000 7fb75effd640 1 -- 172.21.15.83:0/951638589 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb77013be70 con 0x7fb7700a98f0 2024-03-25T19:29:24.814 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.814+0000 7fb75effd640 1 -- 172.21.15.83:0/951638589 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3203347826 0 0) 0x7fb750004eb0 con 0x7fb77013f710 2024-03-25T19:29:24.814 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.814+0000 7fb75effd640 1 -- 172.21.15.83:0/951638589 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7701317f0 con 0x7fb77013f710 2024-03-25T19:29:24.815 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.814+0000 7fb75effd640 1 -- 172.21.15.83:0/951638589 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fb764001b00 con 0x7fb77013f710 2024-03-25T19:29:24.815 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.814+0000 7fb75effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:24.815 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.814+0000 7fb75effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:29:24.815 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.814+0000 7fb75effd640 10 monclient(hunting): dump: 2024-03-25T19:29:24.815 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:29:24.815 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:29:24.815 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:24.815 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:24.815 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:24.815 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:29:24.815 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:29:24.815 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:29:24.815 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:29:24.815 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:24.815 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.814+0000 7fb75effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:24.815 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.814+0000 7fb75effd640 1 -- 172.21.15.83:0/951638589 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2671275989 0 0) 0x7fb770131660 con 0x7fb7701430c0 2024-03-25T19:29:24.815 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.814+0000 7fb75effd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:24.816 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.814+0000 7fb75effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:24.816 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.814+0000 7fb75effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:24.816 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.814+0000 7fb75effd640 10 monclient(hunting): my global_id is 4675 2024-03-25T19:29:24.816 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.814+0000 7fb75effd640 1 -- 172.21.15.83:0/951638589 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb750004eb0 con 0x7fb7701430c0 2024-03-25T19:29:24.825 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.825+0000 7fb75effd640 1 -- 172.21.15.83:0/951638589 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 927550611 0 0) 0x7fb750004eb0 con 0x7fb7701430c0 2024-03-25T19:29:24.825 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.825+0000 7fb75effd640 1 -- 172.21.15.83:0/951638589 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb770131660 con 0x7fb7701430c0 2024-03-25T19:29:24.825 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.825+0000 7fb75effd640 1 -- 172.21.15.83:0/951638589 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fb768001b50 con 0x7fb7701430c0 2024-03-25T19:29:24.825 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.825+0000 7fb75effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:24.825 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.825+0000 7fb75effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:24.825 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.825+0000 7fb75effd640 10 monclient(hunting): dump: 2024-03-25T19:29:24.825 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:29:24.825 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:29:24.825 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:24.825 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:24.826 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:24.826 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:29:24.826 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:29:24.826 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:29:24.826 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:29:24.826 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:24.827 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.826+0000 7fb75effd640 1 -- 172.21.15.83:0/951638589 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2484444148 0 0) 0x7fb7701317f0 con 0x7fb77013f710 2024-03-25T19:29:24.827 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.826+0000 7fb75effd640 1 -- 172.21.15.83:0/951638589 >> v1:172.21.15.83:6790/0 conn(0x7fb7700a98f0 legacy=0x7fb770130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:24.827 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.826+0000 7fb75effd640 1 -- 172.21.15.83:0/951638589 >> v1:172.21.15.103:6789/0 conn(0x7fb7701430c0 legacy=0x7fb7701e2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:24.827 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.826+0000 7fb75effd640 10 monclient: _finish_hunting 0 2024-03-25T19:29:24.827 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.826+0000 7fb75effd640 1 monclient: found mon.a 2024-03-25T19:29:24.827 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.826+0000 7fb75effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:24.827 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.826+0000 7fb75effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:29:24.827 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.826+0000 7fb75effd640 1 -- 172.21.15.83:0/951638589 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb770131980 con 0x7fb77013f710 2024-03-25T19:29:24.828 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.826+0000 7fb75effd640 10 monclient: _finish_auth 0 2024-03-25T19:29:24.828 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.826+0000 7fb75effd640 10 monclient: _check_auth_tickets 2024-03-25T19:29:24.828 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.826+0000 7fb75effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:24.828 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.826+0000 7fb774bc2640 5 monclient: authenticate success, global_id 4659 2024-03-25T19:29:24.828 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.826+0000 7fb774bc2640 10 monclient: _renew_subs 2024-03-25T19:29:24.828 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.826+0000 7fb774bc2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:29:24.828 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.826+0000 7fb774bc2640 1 -- 172.21.15.83:0/951638589 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb770135c60 con 0x7fb77013f710 2024-03-25T19:29:24.828 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.826+0000 7fb774bc2640 10 client.4659.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:24.828 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.826+0000 7fb774bc2640 10 monclient: _renew_subs 2024-03-25T19:29:24.828 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.826+0000 7fb774bc2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:29:24.828 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.826+0000 7fb774bc2640 1 -- 172.21.15.83:0/951638589 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7700b7770 con 0x7fb77013f710 2024-03-25T19:29:24.828 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.826+0000 7fb774bc2640 1 librados: init done 2024-03-25T19:29:24.828 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.827+0000 7fb774bc2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:24.828 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.827+0000 7fb774bc2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:24.828 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.827+0000 7fb774bc2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:29:24.828 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:24.827+0000 7fb774bc2640 1 -- 172.21.15.83:0/951638589 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7701371d0 con 0x7fb77013f710 2024-03-25T19:29:25.143 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:25.143+0000 7fb75effd640 1 -- 172.21.15.83:0/951638589 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb764004a90 con 0x7fb77013f710 2024-03-25T19:29:25.143 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:25.143+0000 7fb75effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:25.143 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:25.143+0000 7fb75effd640 1 -- 172.21.15.83:0/951638589 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fb764004da0 con 0x7fb77013f710 2024-03-25T19:29:25.143 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:25.143+0000 7fb75effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:25.143 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:25.143+0000 7fb75effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:29:25.143 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:25.143+0000 7fb75effd640 10 monclient: dump: 2024-03-25T19:29:25.143 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:29:25.143 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:29:25.143 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:25.143 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:25.143 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:25.143 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:29:25.143 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:29:25.144 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:29:25.144 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:29:25.144 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:25.144 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:25.143+0000 7fb76cd99640 4 set_mon_vals no callback set 2024-03-25T19:29:25.144 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:25.143+0000 7fb75effd640 1 -- 172.21.15.83:0/951638589 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7fb76401ad70 con 0x7fb77013f710 2024-03-25T19:29:25.144 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:25.144+0000 7fb76e59c640 10 monclient: get_auth_request con 0x7fb75006e040 auth_method 0 2024-03-25T19:29:25.144 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:25.144+0000 7fb75effd640 10 client.4659.objecter ms_handle_connect 0x7fb75006e040 2024-03-25T19:29:25.149 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:25.150+0000 7fb75effd640 1 -- 172.21.15.83:0/951638589 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7fb764054870 con 0x7fb77013f710 2024-03-25T19:29:25.150 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:25.150+0000 7fb75effd640 10 client.4659.objecter ms_dispatch 0x7fb770000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:29:25.150 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:25.150+0000 7fb75effd640 3 client.4659.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:29:25.150 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:25.150+0000 7fb75effd640 3 client.4659.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:29:25.150 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:25.150+0000 7fb75effd640 20 client.4659.objecter dump_active .. 0 homeless 2024-03-25T19:29:25.150 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:25.150+0000 7fb75effd640 1 -- 172.21.15.83:0/951638589 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb764050380 con 0x7fb77013f710 2024-03-25T19:29:25.150 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:25.150+0000 7fb75effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:25.150 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:25.150+0000 7fb75effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:25.363 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:25.363+0000 7fb774bc2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:25.363 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:25.363+0000 7fb774bc2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:25.363 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:25.363+0000 7fb774bc2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:29:25.363 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:25.363+0000 7fb774bc2640 1 -- 172.21.15.83:0/951638589 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb770074cc0 con 0x7fb77013f710 2024-03-25T19:29:25.416 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:25.416+0000 7fb75effd640 1 -- 172.21.15.83:0/951638589 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7fb7640522b0 con 0x7fb77013f710 2024-03-25T19:29:25.416 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:25.416+0000 7fb75effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:25.416 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:25.416+0000 7fb75effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:25.416 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:29:25.416 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:29:25.419 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:25.418+0000 7fb774bc2640 10 librados: watch_flush enter 2024-03-25T19:29:25.419 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:25.418+0000 7fb774bc2640 10 librados: watch_flush exit 2024-03-25T19:29:25.419 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:25.419+0000 7fb774bc2640 20 client.4659.objecter shutdown clearing up homeless session... 2024-03-25T19:29:25.419 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:25.419+0000 7fb774bc2640 10 client.4659.objecter successfully canceled tick 2024-03-25T19:29:25.419 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:25.419+0000 7fb774bc2640 1 -- 172.21.15.83:0/951638589 >> v1:172.21.15.103:6800/345072327 conn(0x7fb75006e040 legacy=0x7fb750070500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:25.419 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:25.419+0000 7fb774bc2640 10 monclient: shutdown 2024-03-25T19:29:25.419 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:25.419+0000 7fb774bc2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:25.419 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:25.419+0000 7fb774bc2640 1 -- 172.21.15.83:0/951638589 >> v1:172.21.15.83:6789/0 conn(0x7fb77013f710 legacy=0x7fb7701d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:25.419 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:25.419+0000 7fb774bc2640 1 -- 172.21.15.83:0/951638589 shutdown_connections 2024-03-25T19:29:25.419 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:25.419+0000 7fb774bc2640 1 -- 172.21.15.83:0/951638589 >> 172.21.15.83:0/951638589 conn(0x7fb7700b7ea0 msgr2=0x7fb770074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:25.419 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:25.419+0000 7fb774bc2640 1 -- 172.21.15.83:0/951638589 shutdown_connections 2024-03-25T19:29:25.420 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:25.419+0000 7fb774bc2640 1 -- 172.21.15.83:0/951638589 wait complete. 2024-03-25T19:29:25.420 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:25.419+0000 7fb774bc2640 1 librados: shutdown 2024-03-25T19:29:25.430 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:29:28.972 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:10:10.899 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:28.984 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:10:10.912 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:29.017 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:10:10.945 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:29.032 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:10:10.943 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:29.039 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:10:10.950 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:29.053 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:10:10.981 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:29.061 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:10:10.988 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:29.070 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:10:10.980 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:29.095 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:10:11.022 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:32.237 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:29:32.343 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.344+0000 7ffb6c68d640 10 monclient: get_monmap_and_config 2024-03-25T19:29:32.343 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.344+0000 7ffb6c68d640 10 monclient: build_initial_monmap 2024-03-25T19:29:32.344 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.344+0000 7ffb6c68d640 10 monclient: monmap: 2024-03-25T19:29:32.344 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:29:32.344 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:29:32.344 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:29:32.345178+0000 2024-03-25T19:29:32.344 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:29:32.345178+0000 2024-03-25T19:29:32.344 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:32.344 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:29:32.344 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:29:32.344 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:29:32.344 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:29:32.344 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:32.345 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.344+0000 7ffb6c68d640 1 Processor -- start 2024-03-25T19:29:32.345 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.344+0000 7ffb6c68d640 1 -- start start 2024-03-25T19:29:32.345 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.344+0000 7ffb6c68d640 10 monclient: init 2024-03-25T19:29:32.345 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.345+0000 7ffb6c68d640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:32.346 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.345+0000 7ffb6c68d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:29:32.346 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.345+0000 7ffb6c68d640 10 monclient(hunting): picked mon.noname-b con 0x7ffb640a98f0 addr v1:172.21.15.83:6790/0 2024-03-25T19:29:32.346 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.345+0000 7ffb6c68d640 10 monclient(hunting): picked mon.noname-c con 0x7ffb6413f710 addr v1:172.21.15.103:6789/0 2024-03-25T19:29:32.346 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.345+0000 7ffb6c68d640 10 monclient(hunting): picked mon.noname-a con 0x7ffb641430c0 addr v1:172.21.15.83:6789/0 2024-03-25T19:29:32.346 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.345+0000 7ffb6c68d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb64142520 con 0x7ffb641430c0 2024-03-25T19:29:32.346 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.346+0000 7ffb6c68d640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb640b28a0 con 0x7ffb6413f710 2024-03-25T19:29:32.346 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.346+0000 7ffb6c68d640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb640b2a30 con 0x7ffb640a98f0 2024-03-25T19:29:32.346 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.346+0000 7ffb6c68d640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:32.346 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.346+0000 7ffb6c68d640 10 monclient(hunting): authenticate will time out at 6629.617776s 2024-03-25T19:29:32.354 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.354+0000 7ffb6ac2a640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ffb641430c0 0x7ffb6414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:53342/0 (socket says 172.21.15.83:53342) 2024-03-25T19:29:32.354 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.354+0000 7ffb6ac2a640 1 -- 172.21.15.83:0/3025823307 learned_addr learned my addr 172.21.15.83:0/3025823307 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:29:32.375 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.375+0000 7ffb69427640 1 -- 172.21.15.83:0/3025823307 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3753953816 0 0) 0x7ffb640b2a30 con 0x7ffb640a98f0 2024-03-25T19:29:32.375 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.375+0000 7ffb69427640 10 monclient(hunting): _negotiate 2024-03-25T19:29:32.375 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.375+0000 7ffb69427640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:32.375 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.375+0000 7ffb69427640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:32.375 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.375+0000 7ffb69427640 10 monclient(hunting): my global_id is 4679 2024-03-25T19:29:32.375 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.375+0000 7ffb69427640 1 -- 172.21.15.83:0/3025823307 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb50003520 con 0x7ffb640a98f0 2024-03-25T19:29:32.382 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.382+0000 7ffb69427640 1 -- 172.21.15.83:0/3025823307 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2062519492 0 0) 0x7ffb64142520 con 0x7ffb641430c0 2024-03-25T19:29:32.382 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.382+0000 7ffb69427640 10 monclient(hunting): _negotiate 2024-03-25T19:29:32.382 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.382+0000 7ffb69427640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:32.382 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.382+0000 7ffb69427640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:32.382 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.382+0000 7ffb69427640 10 monclient(hunting): my global_id is 4662 2024-03-25T19:29:32.382 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.382+0000 7ffb69427640 1 -- 172.21.15.83:0/3025823307 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb640b2a30 con 0x7ffb641430c0 2024-03-25T19:29:32.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.386+0000 7ffb69427640 1 -- 172.21.15.83:0/3025823307 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2697278730 0 0) 0x7ffb640b28a0 con 0x7ffb6413f710 2024-03-25T19:29:32.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.386+0000 7ffb69427640 10 monclient(hunting): _negotiate 2024-03-25T19:29:32.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.386+0000 7ffb69427640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:32.386 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.386+0000 7ffb69427640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:32.386 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.386+0000 7ffb69427640 10 monclient(hunting): my global_id is 4678 2024-03-25T19:29:32.386 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.386+0000 7ffb69427640 1 -- 172.21.15.83:0/3025823307 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb64142520 con 0x7ffb6413f710 2024-03-25T19:29:32.395 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.395+0000 7ffb69427640 1 -- 172.21.15.83:0/3025823307 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1450443980 0 0) 0x7ffb50003520 con 0x7ffb640a98f0 2024-03-25T19:29:32.396 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.395+0000 7ffb69427640 1 -- 172.21.15.83:0/3025823307 >> v1:172.21.15.103:6789/0 conn(0x7ffb6413f710 legacy=0x7ffb641429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:32.396 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.395+0000 7ffb69427640 1 -- 172.21.15.83:0/3025823307 >> v1:172.21.15.83:6789/0 conn(0x7ffb641430c0 legacy=0x7ffb6414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:32.396 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.395+0000 7ffb69427640 10 monclient: _finish_hunting 0 2024-03-25T19:29:32.396 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.395+0000 7ffb69427640 1 monclient: found mon.noname-b 2024-03-25T19:29:32.396 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.395+0000 7ffb69427640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:32.396 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.395+0000 7ffb69427640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.83:6790/0 2024-03-25T19:29:32.396 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.395+0000 7ffb69427640 1 -- 172.21.15.83:0/3025823307 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb640b44c0 con 0x7ffb640a98f0 2024-03-25T19:29:32.396 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.395+0000 7ffb69427640 10 monclient: _finish_auth 0 2024-03-25T19:29:32.396 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.395+0000 7ffb69427640 10 monclient: _check_auth_tickets 2024-03-25T19:29:32.396 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.395+0000 7ffb69427640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:32.396 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.395+0000 7ffb6c68d640 5 monclient: authenticate success, global_id 4679 2024-03-25T19:29:32.396 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.395+0000 7ffb6c68d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:32.396 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.395+0000 7ffb69427640 1 -- 172.21.15.83:0/3025823307 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7ffb60001690 con 0x7ffb640a98f0 2024-03-25T19:29:32.396 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.395+0000 7ffb69427640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:32.396 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.395+0000 7ffb69427640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:29:32.397 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.395+0000 7ffb69427640 10 monclient: dump: 2024-03-25T19:29:32.397 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:29:32.397 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:29:32.397 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:32.397 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:32.397 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:32.397 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:29:32.397 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:29:32.397 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:29:32.397 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:29:32.397 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:32.397 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.395+0000 7ffb6c68d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:32.404 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.405+0000 7ffb69427640 1 -- 172.21.15.83:0/3025823307 <== mon.2 v1:172.21.15.83:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb600017f0 con 0x7ffb640a98f0 2024-03-25T19:29:32.405 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.405+0000 7ffb69427640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:32.405 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.405+0000 7ffb69427640 1 -- 172.21.15.83:0/3025823307 <== mon.2 v1:172.21.15.83:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7ffb60001c00 con 0x7ffb640a98f0 2024-03-25T19:29:32.405 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.405+0000 7ffb69427640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:32.405 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.405+0000 7ffb69427640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:29:32.405 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.405+0000 7ffb69427640 10 monclient: dump: 2024-03-25T19:29:32.405 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:29:32.405 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:29:32.405 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:32.405 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:32.405 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:32.406 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:29:32.406 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:29:32.406 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:29:32.406 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:29:32.406 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:32.406 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.405+0000 7ffb6c68d640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:32.406 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.405+0000 7ffb6c68d640 4 set_mon_vals no callback set 2024-03-25T19:29:32.408 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.405+0000 7ffb6c68d640 10 monclient: shutdown 2024-03-25T19:29:32.408 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.405+0000 7ffb6c68d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:32.408 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.405+0000 7ffb6c68d640 1 -- 172.21.15.83:0/3025823307 >> v1:172.21.15.83:6790/0 conn(0x7ffb640a98f0 legacy=0x7ffb640a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:32.408 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.406+0000 7ffb6c68d640 1 -- 172.21.15.83:0/3025823307 shutdown_connections 2024-03-25T19:29:32.408 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.406+0000 7ffb6c68d640 1 -- 172.21.15.83:0/3025823307 >> 172.21.15.83:0/3025823307 conn(0x7ffb640b7ea0 msgr2=0x7ffb640b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:32.409 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.406+0000 7ffb6c68d640 1 -- 172.21.15.83:0/3025823307 shutdown_connections 2024-03-25T19:29:32.409 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.406+0000 7ffb6c68d640 1 -- 172.21.15.83:0/3025823307 wait complete. 2024-03-25T19:29:32.409 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.407+0000 7ffb6c68d640 10 monclient: build_initial_monmap 2024-03-25T19:29:32.409 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.407+0000 7ffb6c68d640 10 monclient: monmap: 2024-03-25T19:29:32.409 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:29:32.409 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:32.409 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:29:32.409 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:29:32.409 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:32.409 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:29:32.409 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:29:32.409 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:29:32.410 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:29:32.410 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:32.410 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.407+0000 7ffb6c68d640 1 librados: starting msgr at 2024-03-25T19:29:32.410 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.407+0000 7ffb6c68d640 1 librados: starting objecter 2024-03-25T19:29:32.410 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.407+0000 7ffb6c68d640 1 Processor -- start 2024-03-25T19:29:32.410 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.407+0000 7ffb6c68d640 1 -- start start 2024-03-25T19:29:32.410 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.407+0000 7ffb6c68d640 1 librados: setting wanted keys 2024-03-25T19:29:32.410 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.407+0000 7ffb6c68d640 1 librados: calling monclient init 2024-03-25T19:29:32.410 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.407+0000 7ffb6c68d640 10 monclient: init 2024-03-25T19:29:32.410 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.408+0000 7ffb6c68d640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:32.410 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.408+0000 7ffb6c68d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:29:32.410 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.408+0000 7ffb6c68d640 10 monclient(hunting): picked mon.noname-a con 0x7ffb640a98f0 addr v1:172.21.15.83:6789/0 2024-03-25T19:29:32.411 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.408+0000 7ffb6c68d640 10 monclient(hunting): picked mon.noname-c con 0x7ffb6413f710 addr v1:172.21.15.83:6790/0 2024-03-25T19:29:32.411 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.408+0000 7ffb6c68d640 10 monclient(hunting): picked mon.noname-b con 0x7ffb641430c0 addr v1:172.21.15.103:6789/0 2024-03-25T19:29:32.411 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.408+0000 7ffb6c68d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb64138b20 con 0x7ffb640a98f0 2024-03-25T19:29:32.411 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.408+0000 7ffb6c68d640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb64137a80 con 0x7ffb641430c0 2024-03-25T19:29:32.411 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.408+0000 7ffb6c68d640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb64139880 con 0x7ffb6413f710 2024-03-25T19:29:32.411 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.408+0000 7ffb6c68d640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:32.411 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.408+0000 7ffb6c68d640 10 monclient(hunting): authenticate will time out at 6629.680544s 2024-03-25T19:29:32.415 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.416+0000 7ffb6ac2a640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7ffb641430c0 0x7ffb641e2040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:46588/0 (socket says 172.21.15.83:46588) 2024-03-25T19:29:32.415 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.416+0000 7ffb6ac2a640 1 -- 172.21.15.83:0/981176922 learned_addr learned my addr 172.21.15.83:0/981176922 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:29:32.417 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.417+0000 7ffb5affd640 10 client.?.objecter ms_handle_connect 0x7ffb641430c0 2024-03-25T19:29:32.417 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.417+0000 7ffb5affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:32.422 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.423+0000 7ffb5affd640 10 client.?.objecter ms_handle_connect 0x7ffb640a98f0 2024-03-25T19:29:32.422 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.423+0000 7ffb5affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:32.429 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.429+0000 7ffb5affd640 1 -- 172.21.15.83:0/981176922 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1914577453 0 0) 0x7ffb64137a80 con 0x7ffb641430c0 2024-03-25T19:29:32.429 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.429+0000 7ffb5affd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:32.429 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.429+0000 7ffb5affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:32.429 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.429+0000 7ffb5affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:32.429 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.429+0000 7ffb5affd640 10 monclient(hunting): my global_id is 4681 2024-03-25T19:29:32.429 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.429+0000 7ffb5affd640 1 -- 172.21.15.83:0/981176922 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb48004eb0 con 0x7ffb641430c0 2024-03-25T19:29:32.435 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.435+0000 7ffb5affd640 1 -- 172.21.15.83:0/981176922 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 604816266 0 0) 0x7ffb64138b20 con 0x7ffb640a98f0 2024-03-25T19:29:32.435 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.435+0000 7ffb5affd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:32.435 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.435+0000 7ffb5affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:32.435 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.435+0000 7ffb5affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:32.435 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.435+0000 7ffb5affd640 10 monclient(hunting): my global_id is 4665 2024-03-25T19:29:32.435 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.435+0000 7ffb5affd640 1 -- 172.21.15.83:0/981176922 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb64137a80 con 0x7ffb640a98f0 2024-03-25T19:29:32.436 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.437+0000 7ffb5affd640 10 client.?.objecter ms_handle_connect 0x7ffb6413f710 2024-03-25T19:29:32.436 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.437+0000 7ffb5affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:32.441 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.441+0000 7ffb5affd640 1 -- 172.21.15.83:0/981176922 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 600772274 0 0) 0x7ffb48004eb0 con 0x7ffb641430c0 2024-03-25T19:29:32.441 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.441+0000 7ffb5affd640 1 -- 172.21.15.83:0/981176922 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb64138b20 con 0x7ffb641430c0 2024-03-25T19:29:32.441 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.441+0000 7ffb5affd640 1 -- 172.21.15.83:0/981176922 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7ffb5c001c70 con 0x7ffb641430c0 2024-03-25T19:29:32.441 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.441+0000 7ffb5affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:32.441 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.441+0000 7ffb5affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:29:32.441 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.441+0000 7ffb5affd640 10 monclient(hunting): dump: 2024-03-25T19:29:32.441 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:29:32.441 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:29:32.442 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:32.442 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:32.442 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:32.442 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:29:32.442 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:29:32.442 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:29:32.442 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:29:32.442 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:32.442 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.441+0000 7ffb5affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:32.449 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.449+0000 7ffb5affd640 1 -- 172.21.15.83:0/981176922 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2996065431 0 0) 0x7ffb64137a80 con 0x7ffb640a98f0 2024-03-25T19:29:32.449 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.449+0000 7ffb5affd640 1 -- 172.21.15.83:0/981176922 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb48004eb0 con 0x7ffb640a98f0 2024-03-25T19:29:32.449 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.449+0000 7ffb5affd640 1 -- 172.21.15.83:0/981176922 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7ffb60001b00 con 0x7ffb640a98f0 2024-03-25T19:29:32.449 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.449+0000 7ffb5affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:32.449 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.449+0000 7ffb5affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:29:32.449 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.449+0000 7ffb5affd640 10 monclient(hunting): dump: 2024-03-25T19:29:32.449 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:29:32.449 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:29:32.449 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:32.449 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:32.449 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:32.449 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:29:32.450 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:29:32.450 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:29:32.450 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:29:32.450 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:32.450 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.451+0000 7ffb5affd640 1 -- 172.21.15.83:0/981176922 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1956632705 0 0) 0x7ffb64139880 con 0x7ffb6413f710 2024-03-25T19:29:32.450 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.451+0000 7ffb5affd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:32.450 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.451+0000 7ffb5affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:32.451 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.451+0000 7ffb5affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:32.451 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.451+0000 7ffb5affd640 10 monclient(hunting): my global_id is 4682 2024-03-25T19:29:32.451 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.451+0000 7ffb5affd640 1 -- 172.21.15.83:0/981176922 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb64137a80 con 0x7ffb6413f710 2024-03-25T19:29:32.464 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.464+0000 7ffb5affd640 1 -- 172.21.15.83:0/981176922 <== mon.1 v1:172.21.15.103:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1025257742 0 0) 0x7ffb64138b20 con 0x7ffb641430c0 2024-03-25T19:29:32.464 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.464+0000 7ffb5affd640 1 -- 172.21.15.83:0/981176922 >> v1:172.21.15.83:6790/0 conn(0x7ffb6413f710 legacy=0x7ffb64139170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:32.464 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.464+0000 7ffb5affd640 1 -- 172.21.15.83:0/981176922 >> v1:172.21.15.83:6789/0 conn(0x7ffb640a98f0 legacy=0x7ffb64137370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:32.464 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.464+0000 7ffb5affd640 10 monclient: _finish_hunting 0 2024-03-25T19:29:32.464 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.464+0000 7ffb5affd640 1 monclient: found mon.b 2024-03-25T19:29:32.464 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.464+0000 7ffb5affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:32.464 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.464+0000 7ffb5affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:29:32.464 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.464+0000 7ffb5affd640 1 -- 172.21.15.83:0/981176922 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb64139a10 con 0x7ffb641430c0 2024-03-25T19:29:32.464 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.464+0000 7ffb5affd640 10 monclient: _finish_auth 0 2024-03-25T19:29:32.464 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.464+0000 7ffb5affd640 10 monclient: _check_auth_tickets 2024-03-25T19:29:32.464 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.464+0000 7ffb5affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:32.465 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.464+0000 7ffb6c68d640 5 monclient: authenticate success, global_id 4681 2024-03-25T19:29:32.465 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.464+0000 7ffb6c68d640 10 monclient: _renew_subs 2024-03-25T19:29:32.465 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.464+0000 7ffb6c68d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:29:32.465 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.464+0000 7ffb6c68d640 1 -- 172.21.15.83:0/981176922 --> v1:172.21.15.103:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb641dab00 con 0x7ffb641430c0 2024-03-25T19:29:32.465 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.464+0000 7ffb6c68d640 10 client.4681.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:32.465 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.464+0000 7ffb6c68d640 10 monclient: _renew_subs 2024-03-25T19:29:32.465 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.464+0000 7ffb6c68d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:29:32.465 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.464+0000 7ffb6c68d640 1 -- 172.21.15.83:0/981176922 --> v1:172.21.15.103:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb6413b4f0 con 0x7ffb641430c0 2024-03-25T19:29:32.465 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.464+0000 7ffb6c68d640 1 librados: init done 2024-03-25T19:29:32.465 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.465+0000 7ffb6c68d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:32.466 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.465+0000 7ffb6c68d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:32.466 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.465+0000 7ffb6c68d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:29:32.466 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.465+0000 7ffb6c68d640 1 -- 172.21.15.83:0/981176922 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb641dae00 con 0x7ffb641430c0 2024-03-25T19:29:32.757 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.757+0000 7ffb5affd640 1 -- 172.21.15.83:0/981176922 <== mon.1 v1:172.21.15.103:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb5c004b60 con 0x7ffb641430c0 2024-03-25T19:29:32.757 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.757+0000 7ffb5affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:32.757 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.757+0000 7ffb5affd640 1 -- 172.21.15.83:0/981176922 <== mon.1 v1:172.21.15.103:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7ffb5c004e70 con 0x7ffb641430c0 2024-03-25T19:29:32.757 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.757+0000 7ffb5affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:32.757 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.757+0000 7ffb5affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:32.757 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.757+0000 7ffb5affd640 10 monclient: dump: 2024-03-25T19:29:32.757 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:29:32.757 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:29:32.757 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:32.757 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:32.757 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:32.757 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:29:32.757 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:29:32.758 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:29:32.758 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:29:32.758 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:32.758 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.757+0000 7ffb68c26640 4 set_mon_vals no callback set 2024-03-25T19:29:32.758 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.757+0000 7ffb5affd640 1 -- 172.21.15.83:0/981176922 <== mon.1 v1:172.21.15.103:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7ffb5c01ae40 con 0x7ffb641430c0 2024-03-25T19:29:32.758 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.758+0000 7ffb6a429640 10 monclient: get_auth_request con 0x7ffb4806dfe0 auth_method 0 2024-03-25T19:29:32.758 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.758+0000 7ffb5affd640 10 client.4681.objecter ms_handle_connect 0x7ffb4806dfe0 2024-03-25T19:29:32.762 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.763+0000 7ffb5affd640 1 -- 172.21.15.83:0/981176922 <== mon.1 v1:172.21.15.103:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7ffb5c054b80 con 0x7ffb641430c0 2024-03-25T19:29:32.762 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.763+0000 7ffb5affd640 10 client.4681.objecter ms_dispatch 0x7ffb64000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:29:32.762 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.763+0000 7ffb5affd640 3 client.4681.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:29:32.763 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.763+0000 7ffb5affd640 3 client.4681.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:29:32.763 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.763+0000 7ffb5affd640 20 client.4681.objecter dump_active .. 0 homeless 2024-03-25T19:29:32.764 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.764+0000 7ffb5affd640 1 -- 172.21.15.83:0/981176922 <== mon.1 v1:172.21.15.103:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffb5c0504c0 con 0x7ffb641430c0 2024-03-25T19:29:32.764 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.764+0000 7ffb5affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:32.764 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:32.764+0000 7ffb5affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:33.010 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:33.011+0000 7ffb6c68d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:33.011 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:33.011+0000 7ffb6c68d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:33.011 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:33.011+0000 7ffb6c68d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:29:33.011 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:33.011+0000 7ffb6c68d640 1 -- 172.21.15.83:0/981176922 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb640b3e80 con 0x7ffb641430c0 2024-03-25T19:29:33.045 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:33.046+0000 7ffb5affd640 1 -- 172.21.15.83:0/981176922 <== mon.1 v1:172.21.15.103:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7ffb5c0525c0 con 0x7ffb641430c0 2024-03-25T19:29:33.046 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:33.046+0000 7ffb5affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:33.046 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:33.046+0000 7ffb5affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:33.046 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:29:33.046 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:29:33.048 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:33.048+0000 7ffb6c68d640 10 librados: watch_flush enter 2024-03-25T19:29:33.048 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:33.048+0000 7ffb6c68d640 10 librados: watch_flush exit 2024-03-25T19:29:33.048 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:33.048+0000 7ffb6c68d640 20 client.4681.objecter shutdown clearing up homeless session... 2024-03-25T19:29:33.048 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:33.048+0000 7ffb6c68d640 10 client.4681.objecter successfully canceled tick 2024-03-25T19:29:33.048 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:33.048+0000 7ffb6c68d640 1 -- 172.21.15.83:0/981176922 >> v1:172.21.15.103:6800/345072327 conn(0x7ffb4806dfe0 legacy=0x7ffb480704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:33.049 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:33.048+0000 7ffb6c68d640 10 monclient: shutdown 2024-03-25T19:29:33.049 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:33.048+0000 7ffb6c68d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:33.049 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:33.048+0000 7ffb6c68d640 1 -- 172.21.15.83:0/981176922 >> v1:172.21.15.103:6789/0 conn(0x7ffb641430c0 legacy=0x7ffb641e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:33.049 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:33.048+0000 7ffb6c68d640 1 -- 172.21.15.83:0/981176922 shutdown_connections 2024-03-25T19:29:33.049 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:33.048+0000 7ffb6c68d640 1 -- 172.21.15.83:0/981176922 >> 172.21.15.83:0/981176922 conn(0x7ffb640b7ea0 msgr2=0x7ffb64074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:33.049 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:33.048+0000 7ffb6c68d640 1 -- 172.21.15.83:0/981176922 shutdown_connections 2024-03-25T19:29:33.049 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:33.048+0000 7ffb6c68d640 1 -- 172.21.15.83:0/981176922 wait complete. 2024-03-25T19:29:33.049 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:33.048+0000 7ffb6c68d640 1 librados: shutdown 2024-03-25T19:29:33.060 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:29:39.866 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:29:39.981 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:39.981+0000 7f01f16d6640 10 monclient: get_monmap_and_config 2024-03-25T19:29:39.981 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:39.981+0000 7f01f16d6640 10 monclient: build_initial_monmap 2024-03-25T19:29:39.981 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:39.981+0000 7f01f16d6640 10 monclient: monmap: 2024-03-25T19:29:39.981 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:29:39.981 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:29:39.981 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:29:39.982546+0000 2024-03-25T19:29:39.981 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:29:39.982546+0000 2024-03-25T19:29:39.982 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:39.982 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:29:39.982 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:29:39.982 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:29:39.982 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:29:39.982 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:39.983 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:39.982+0000 7f01f16d6640 1 Processor -- start 2024-03-25T19:29:39.983 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:39.982+0000 7f01f16d6640 1 -- start start 2024-03-25T19:29:39.983 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:39.982+0000 7f01f16d6640 10 monclient: init 2024-03-25T19:29:39.983 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:39.982+0000 7f01f16d6640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:39.983 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:39.982+0000 7f01f16d6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:29:39.983 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:39.982+0000 7f01f16d6640 10 monclient(hunting): picked mon.noname-a con 0x7f01ec0a98f0 addr v1:172.21.15.83:6789/0 2024-03-25T19:29:39.983 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:39.982+0000 7f01f16d6640 10 monclient(hunting): picked mon.noname-c con 0x7f01ec13f710 addr v1:172.21.15.103:6789/0 2024-03-25T19:29:39.983 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:39.982+0000 7f01f16d6640 10 monclient(hunting): picked mon.noname-b con 0x7f01ec1430c0 addr v1:172.21.15.83:6790/0 2024-03-25T19:29:39.984 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:39.982+0000 7f01f16d6640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01ec142520 con 0x7f01ec0a98f0 2024-03-25T19:29:39.984 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:39.982+0000 7f01f16d6640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01ec0b28a0 con 0x7f01ec13f710 2024-03-25T19:29:39.984 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:39.982+0000 7f01f16d6640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01ec0b2a30 con 0x7f01ec1430c0 2024-03-25T19:29:39.984 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:39.982+0000 7f01f16d6640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:39.984 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:39.982+0000 7f01f16d6640 10 monclient(hunting): authenticate will time out at 6637.254726s 2024-03-25T19:29:39.994 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:39.995+0000 7f01eb7fe640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f01ec1430c0 0x7f01ec14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:47860/0 (socket says 172.21.15.83:47860) 2024-03-25T19:29:39.994 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:39.995+0000 7f01eb7fe640 1 -- 172.21.15.83:0/1341220769 learned_addr learned my addr 172.21.15.83:0/1341220769 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:29:40.008 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.009+0000 7f01e9ffb640 1 -- 172.21.15.83:0/1341220769 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2592307326 0 0) 0x7f01ec0b2a30 con 0x7f01ec1430c0 2024-03-25T19:29:40.008 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.009+0000 7f01e9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:29:40.008 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.009+0000 7f01e9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:40.008 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.009+0000 7f01e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:40.008 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.009+0000 7f01e9ffb640 10 monclient(hunting): my global_id is 4685 2024-03-25T19:29:40.008 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.009+0000 7f01e9ffb640 1 -- 172.21.15.83:0/1341220769 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01d4003520 con 0x7f01ec1430c0 2024-03-25T19:29:40.010 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.010+0000 7f01e9ffb640 1 -- 172.21.15.83:0/1341220769 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1740012295 0 0) 0x7f01ec0b28a0 con 0x7f01ec13f710 2024-03-25T19:29:40.010 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.010+0000 7f01e9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:29:40.010 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.010+0000 7f01e9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:40.010 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.010+0000 7f01e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:40.010 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.010+0000 7f01e9ffb640 10 monclient(hunting): my global_id is 4684 2024-03-25T19:29:40.010 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.010+0000 7f01e9ffb640 1 -- 172.21.15.83:0/1341220769 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01ec0b2a30 con 0x7f01ec13f710 2024-03-25T19:29:40.013 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.014+0000 7f01e9ffb640 1 -- 172.21.15.83:0/1341220769 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2765884779 0 0) 0x7f01ec142520 con 0x7f01ec0a98f0 2024-03-25T19:29:40.014 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.014+0000 7f01e9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:29:40.014 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.014+0000 7f01e9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:40.014 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.014+0000 7f01e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:40.014 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.014+0000 7f01e9ffb640 10 monclient(hunting): my global_id is 4668 2024-03-25T19:29:40.014 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.014+0000 7f01e9ffb640 1 -- 172.21.15.83:0/1341220769 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01ec0b28a0 con 0x7f01ec0a98f0 2024-03-25T19:29:40.026 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.026+0000 7f01e9ffb640 1 -- 172.21.15.83:0/1341220769 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2163176317 0 0) 0x7f01d4003520 con 0x7f01ec1430c0 2024-03-25T19:29:40.026 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.026+0000 7f01e9ffb640 1 -- 172.21.15.83:0/1341220769 >> v1:172.21.15.103:6789/0 conn(0x7f01ec13f710 legacy=0x7f01ec1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:40.026 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.026+0000 7f01e9ffb640 1 -- 172.21.15.83:0/1341220769 >> v1:172.21.15.83:6789/0 conn(0x7f01ec0a98f0 legacy=0x7f01ec0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:40.026 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.026+0000 7f01e9ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:29:40.026 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.026+0000 7f01e9ffb640 1 monclient: found mon.noname-b 2024-03-25T19:29:40.026 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.026+0000 7f01e9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:40.026 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.026+0000 7f01e9ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.83:6790/0 2024-03-25T19:29:40.026 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.026+0000 7f01e9ffb640 1 -- 172.21.15.83:0/1341220769 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01ec0b44c0 con 0x7f01ec1430c0 2024-03-25T19:29:40.026 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.026+0000 7f01e9ffb640 10 monclient: _finish_auth 0 2024-03-25T19:29:40.026 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.026+0000 7f01e9ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:29:40.026 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.026+0000 7f01e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:40.027 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.026+0000 7f01e9ffb640 1 -- 172.21.15.83:0/1341220769 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f01e4001a10 con 0x7f01ec1430c0 2024-03-25T19:29:40.027 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.026+0000 7f01e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:40.027 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.026+0000 7f01e9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:29:40.027 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.026+0000 7f01e9ffb640 10 monclient: dump: 2024-03-25T19:29:40.027 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:29:40.027 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:29:40.027 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:40.027 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:40.027 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:40.027 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:29:40.027 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:29:40.027 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:29:40.027 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:29:40.027 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:40.027 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.026+0000 7f01f16d6640 5 monclient: authenticate success, global_id 4685 2024-03-25T19:29:40.028 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.026+0000 7f01f16d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:40.055 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.055+0000 7f01e9ffb640 1 -- 172.21.15.83:0/1341220769 <== mon.2 v1:172.21.15.83:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f01e4001b70 con 0x7f01ec1430c0 2024-03-25T19:29:40.055 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.055+0000 7f01e9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:40.055 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.055+0000 7f01e9ffb640 1 -- 172.21.15.83:0/1341220769 <== mon.2 v1:172.21.15.83:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f01e4001f80 con 0x7f01ec1430c0 2024-03-25T19:29:40.055 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.055+0000 7f01e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:40.055 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.055+0000 7f01e9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:29:40.055 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.055+0000 7f01e9ffb640 10 monclient: dump: 2024-03-25T19:29:40.055 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:29:40.055 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:29:40.056 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:40.056 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:40.056 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:40.056 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:29:40.056 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:29:40.056 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:29:40.056 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:29:40.056 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:40.056 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.055+0000 7f01f16d6640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:40.056 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.055+0000 7f01f16d6640 4 set_mon_vals no callback set 2024-03-25T19:29:40.057 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.056+0000 7f01f16d6640 10 monclient: shutdown 2024-03-25T19:29:40.057 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.056+0000 7f01f16d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:40.057 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.056+0000 7f01f16d6640 1 -- 172.21.15.83:0/1341220769 >> v1:172.21.15.83:6790/0 conn(0x7f01ec1430c0 legacy=0x7f01ec14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:40.057 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.056+0000 7f01f16d6640 1 -- 172.21.15.83:0/1341220769 shutdown_connections 2024-03-25T19:29:40.057 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.056+0000 7f01f16d6640 1 -- 172.21.15.83:0/1341220769 >> 172.21.15.83:0/1341220769 conn(0x7f01ec0b7ea0 msgr2=0x7f01ec0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:40.058 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.056+0000 7f01f16d6640 1 -- 172.21.15.83:0/1341220769 shutdown_connections 2024-03-25T19:29:40.058 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.056+0000 7f01f16d6640 1 -- 172.21.15.83:0/1341220769 wait complete. 2024-03-25T19:29:40.058 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.057+0000 7f01f16d6640 10 monclient: build_initial_monmap 2024-03-25T19:29:40.058 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.057+0000 7f01f16d6640 10 monclient: monmap: 2024-03-25T19:29:40.058 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:29:40.058 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:40.058 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:29:40.058 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:29:40.058 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:40.058 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:29:40.058 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:29:40.058 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:29:40.058 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:29:40.058 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:40.058 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.058+0000 7f01f16d6640 1 librados: starting msgr at 2024-03-25T19:29:40.059 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.058+0000 7f01f16d6640 1 librados: starting objecter 2024-03-25T19:29:40.059 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.058+0000 7f01f16d6640 1 Processor -- start 2024-03-25T19:29:40.060 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.058+0000 7f01f16d6640 1 -- start start 2024-03-25T19:29:40.060 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.058+0000 7f01f16d6640 1 librados: setting wanted keys 2024-03-25T19:29:40.060 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.058+0000 7f01f16d6640 1 librados: calling monclient init 2024-03-25T19:29:40.060 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.058+0000 7f01f16d6640 10 monclient: init 2024-03-25T19:29:40.060 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.059+0000 7f01f16d6640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:40.060 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.059+0000 7f01f16d6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:29:40.060 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.059+0000 7f01f16d6640 10 monclient(hunting): picked mon.noname-a con 0x7f01ec0a98f0 addr v1:172.21.15.83:6789/0 2024-03-25T19:29:40.060 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.059+0000 7f01f16d6640 10 monclient(hunting): picked mon.noname-b con 0x7f01ec13f710 addr v1:172.21.15.103:6789/0 2024-03-25T19:29:40.060 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.059+0000 7f01f16d6640 10 monclient(hunting): picked mon.noname-c con 0x7f01ec1430c0 addr v1:172.21.15.83:6790/0 2024-03-25T19:29:40.060 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.059+0000 7f01f16d6640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01ec138a70 con 0x7f01ec0a98f0 2024-03-25T19:29:40.060 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.059+0000 7f01f16d6640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01ec0aba50 con 0x7f01ec13f710 2024-03-25T19:29:40.060 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.059+0000 7f01f16d6640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01ec1d77a0 con 0x7f01ec1430c0 2024-03-25T19:29:40.060 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.059+0000 7f01f16d6640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:40.060 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.059+0000 7f01f16d6640 10 monclient(hunting): authenticate will time out at 6637.331249s 2024-03-25T19:29:40.066 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.067+0000 7f01eaffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f01ec0a98f0 0x7f01ec138360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:45960/0 (socket says 172.21.15.83:45960) 2024-03-25T19:29:40.067 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.067+0000 7f01eaffd640 1 -- 172.21.15.83:0/3257493517 learned_addr learned my addr 172.21.15.83:0/3257493517 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:29:40.068 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.068+0000 7f01d37fe640 10 client.?.objecter ms_handle_connect 0x7f01ec0a98f0 2024-03-25T19:29:40.068 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.068+0000 7f01d37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:40.079 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.079+0000 7f01d37fe640 10 client.?.objecter ms_handle_connect 0x7f01ec13f710 2024-03-25T19:29:40.079 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.079+0000 7f01d37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:40.079 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.079+0000 7f01d37fe640 10 client.?.objecter ms_handle_connect 0x7f01ec1430c0 2024-03-25T19:29:40.079 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.079+0000 7f01d37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:40.081 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.082+0000 7f01d37fe640 1 -- 172.21.15.83:0/3257493517 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 886122724 0 0) 0x7f01ec138a70 con 0x7f01ec0a98f0 2024-03-25T19:29:40.081 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.082+0000 7f01d37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:29:40.081 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.082+0000 7f01d37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:40.081 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.082+0000 7f01d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:40.082 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.082+0000 7f01d37fe640 10 monclient(hunting): my global_id is 4671 2024-03-25T19:29:40.082 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.082+0000 7f01d37fe640 1 -- 172.21.15.83:0/3257493517 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01d8004eb0 con 0x7f01ec0a98f0 2024-03-25T19:29:40.088 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.088+0000 7f01d37fe640 1 -- 172.21.15.83:0/3257493517 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3447798895 0 0) 0x7f01ec0aba50 con 0x7f01ec13f710 2024-03-25T19:29:40.088 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.088+0000 7f01d37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:29:40.088 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.088+0000 7f01d37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:40.088 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.088+0000 7f01d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:40.088 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.088+0000 7f01d37fe640 10 monclient(hunting): my global_id is 4687 2024-03-25T19:29:40.088 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.088+0000 7f01d37fe640 1 -- 172.21.15.83:0/3257493517 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01ec138a70 con 0x7f01ec13f710 2024-03-25T19:29:40.095 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.096+0000 7f01d37fe640 1 -- 172.21.15.83:0/3257493517 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2467098849 0 0) 0x7f01ec1d77a0 con 0x7f01ec1430c0 2024-03-25T19:29:40.096 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.096+0000 7f01d37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:29:40.096 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.096+0000 7f01d37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:40.096 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.096+0000 7f01d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:40.096 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.096+0000 7f01d37fe640 10 monclient(hunting): my global_id is 4688 2024-03-25T19:29:40.096 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.096+0000 7f01d37fe640 1 -- 172.21.15.83:0/3257493517 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01ec0aba50 con 0x7f01ec1430c0 2024-03-25T19:29:40.098 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.099+0000 7f01d37fe640 1 -- 172.21.15.83:0/3257493517 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 270880792 0 0) 0x7f01d8004eb0 con 0x7f01ec0a98f0 2024-03-25T19:29:40.098 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.099+0000 7f01d37fe640 1 -- 172.21.15.83:0/3257493517 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f01ec1d77a0 con 0x7f01ec0a98f0 2024-03-25T19:29:40.099 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.099+0000 7f01d37fe640 1 -- 172.21.15.83:0/3257493517 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f01dc001a10 con 0x7f01ec0a98f0 2024-03-25T19:29:40.099 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.099+0000 7f01d37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:40.099 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.099+0000 7f01d37fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:29:40.099 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.099+0000 7f01d37fe640 10 monclient(hunting): dump: 2024-03-25T19:29:40.099 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:29:40.099 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:29:40.099 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:40.099 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:40.099 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:40.099 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:29:40.099 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:29:40.099 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:29:40.099 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:29:40.099 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:40.099 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.099+0000 7f01d37fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:40.116 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.115+0000 7f01d37fe640 1 -- 172.21.15.83:0/3257493517 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1200043757 0 0) 0x7f01ec138a70 con 0x7f01ec13f710 2024-03-25T19:29:40.116 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.115+0000 7f01d37fe640 1 -- 172.21.15.83:0/3257493517 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f01d8004eb0 con 0x7f01ec13f710 2024-03-25T19:29:40.116 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.116+0000 7f01d37fe640 1 -- 172.21.15.83:0/3257493517 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f01e0001970 con 0x7f01ec13f710 2024-03-25T19:29:40.116 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.116+0000 7f01d37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:40.116 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.116+0000 7f01d37fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:40.116 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.116+0000 7f01d37fe640 10 monclient(hunting): dump: 2024-03-25T19:29:40.117 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:29:40.117 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:29:40.117 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:40.117 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:40.117 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:40.117 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:29:40.117 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:29:40.117 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:29:40.117 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:29:40.117 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:40.117 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.116+0000 7f01d37fe640 1 -- 172.21.15.83:0/3257493517 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 420800058 0 0) 0x7f01ec1d77a0 con 0x7f01ec0a98f0 2024-03-25T19:29:40.117 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.116+0000 7f01d37fe640 1 -- 172.21.15.83:0/3257493517 >> v1:172.21.15.83:6790/0 conn(0x7f01ec1430c0 legacy=0x7f01ec1d4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:40.117 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.116+0000 7f01d37fe640 1 -- 172.21.15.83:0/3257493517 >> v1:172.21.15.103:6789/0 conn(0x7f01ec13f710 legacy=0x7f01ec1d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:40.117 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.116+0000 7f01d37fe640 10 monclient: _finish_hunting 0 2024-03-25T19:29:40.117 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.116+0000 7f01d37fe640 1 monclient: found mon.a 2024-03-25T19:29:40.117 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.116+0000 7f01d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:40.117 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.116+0000 7f01d37fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:29:40.118 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.116+0000 7f01d37fe640 1 -- 172.21.15.83:0/3257493517 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01ec1d83d0 con 0x7f01ec0a98f0 2024-03-25T19:29:40.118 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.116+0000 7f01d37fe640 10 monclient: _finish_auth 0 2024-03-25T19:29:40.118 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.116+0000 7f01d37fe640 10 monclient: _check_auth_tickets 2024-03-25T19:29:40.118 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.116+0000 7f01d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:40.118 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.116+0000 7f01f16d6640 5 monclient: authenticate success, global_id 4671 2024-03-25T19:29:40.118 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.116+0000 7f01f16d6640 10 monclient: _renew_subs 2024-03-25T19:29:40.118 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.116+0000 7f01f16d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:29:40.118 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.116+0000 7f01f16d6640 1 -- 172.21.15.83:0/3257493517 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01ec1d41e0 con 0x7f01ec0a98f0 2024-03-25T19:29:40.118 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.116+0000 7f01f16d6640 10 client.4671.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:40.118 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.116+0000 7f01f16d6640 10 monclient: _renew_subs 2024-03-25T19:29:40.118 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.116+0000 7f01f16d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:29:40.118 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.116+0000 7f01f16d6640 1 -- 172.21.15.83:0/3257493517 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f01ec1d5110 con 0x7f01ec0a98f0 2024-03-25T19:29:40.118 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.116+0000 7f01f16d6640 1 librados: init done 2024-03-25T19:29:40.119 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.117+0000 7f01f16d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:40.119 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.117+0000 7f01f16d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:40.119 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.117+0000 7f01f16d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:29:40.119 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.117+0000 7f01f16d6640 1 -- 172.21.15.83:0/3257493517 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01ec14f320 con 0x7f01ec0a98f0 2024-03-25T19:29:40.426 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.426+0000 7f01d37fe640 1 -- 172.21.15.83:0/3257493517 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f01dc001b70 con 0x7f01ec0a98f0 2024-03-25T19:29:40.426 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.426+0000 7f01d37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:40.426 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.426+0000 7f01d37fe640 1 -- 172.21.15.83:0/3257493517 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f01dc003a60 con 0x7f01ec0a98f0 2024-03-25T19:29:40.427 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.426+0000 7f01d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:40.427 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.426+0000 7f01d37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:29:40.427 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.426+0000 7f01d37fe640 10 monclient: dump: 2024-03-25T19:29:40.427 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:29:40.427 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:29:40.427 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:40.427 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:40.427 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:40.427 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:29:40.427 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:29:40.427 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:29:40.427 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:29:40.427 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:40.427 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.426+0000 7f01e97fa640 4 set_mon_vals no callback set 2024-03-25T19:29:40.428 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.427+0000 7f01d37fe640 1 -- 172.21.15.83:0/3257493517 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7f01dc003c40 con 0x7f01ec0a98f0 2024-03-25T19:29:40.428 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.428+0000 7f01ea7fc640 10 monclient: get_auth_request con 0x7f01d806df70 auth_method 0 2024-03-25T19:29:40.428 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.428+0000 7f01d37fe640 10 client.4671.objecter ms_handle_connect 0x7f01d806df70 2024-03-25T19:29:40.433 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.434+0000 7f01d37fe640 1 -- 172.21.15.83:0/3257493517 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7f01dc054940 con 0x7f01ec0a98f0 2024-03-25T19:29:40.433 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.434+0000 7f01d37fe640 10 client.4671.objecter ms_dispatch 0x7f01ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:29:40.434 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.434+0000 7f01d37fe640 3 client.4671.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:29:40.434 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.434+0000 7f01d37fe640 3 client.4671.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:29:40.434 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.434+0000 7f01d37fe640 20 client.4671.objecter dump_active .. 0 homeless 2024-03-25T19:29:40.434 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.434+0000 7f01d37fe640 1 -- 172.21.15.83:0/3257493517 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f01dc050280 con 0x7f01ec0a98f0 2024-03-25T19:29:40.434 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.434+0000 7f01d37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:40.434 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.434+0000 7f01d37fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:40.660 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.661+0000 7f01f16d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:40.661 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.661+0000 7f01f16d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:40.661 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.661+0000 7f01f16d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:29:40.661 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.661+0000 7f01f16d6640 1 -- 172.21.15.83:0/3257493517 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f01ec1d7930 con 0x7f01ec0a98f0 2024-03-25T19:29:40.701 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.701+0000 7f01d37fe640 1 -- 172.21.15.83:0/3257493517 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7f01dc052380 con 0x7f01ec0a98f0 2024-03-25T19:29:40.701 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.701+0000 7f01d37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:40.701 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.701+0000 7f01d37fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:40.701 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:29:40.702 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:29:40.704 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.703+0000 7f01f16d6640 10 librados: watch_flush enter 2024-03-25T19:29:40.704 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.703+0000 7f01f16d6640 10 librados: watch_flush exit 2024-03-25T19:29:40.704 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.703+0000 7f01f16d6640 20 client.4671.objecter shutdown clearing up homeless session... 2024-03-25T19:29:40.704 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.703+0000 7f01f16d6640 10 client.4671.objecter successfully canceled tick 2024-03-25T19:29:40.704 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.703+0000 7f01f16d6640 1 -- 172.21.15.83:0/3257493517 >> v1:172.21.15.103:6800/345072327 conn(0x7f01d806df70 legacy=0x7f01d8070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:40.704 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.703+0000 7f01f16d6640 10 monclient: shutdown 2024-03-25T19:29:40.704 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.703+0000 7f01f16d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:40.704 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.703+0000 7f01f16d6640 1 -- 172.21.15.83:0/3257493517 >> v1:172.21.15.83:6789/0 conn(0x7f01ec0a98f0 legacy=0x7f01ec138360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:40.704 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.704+0000 7f01f16d6640 1 -- 172.21.15.83:0/3257493517 shutdown_connections 2024-03-25T19:29:40.704 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.704+0000 7f01f16d6640 1 -- 172.21.15.83:0/3257493517 >> 172.21.15.83:0/3257493517 conn(0x7f01ec0b7ea0 msgr2=0x7f01ec13ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:40.705 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.704+0000 7f01f16d6640 1 -- 172.21.15.83:0/3257493517 shutdown_connections 2024-03-25T19:29:40.705 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.704+0000 7f01f16d6640 1 -- 172.21.15.83:0/3257493517 wait complete. 2024-03-25T19:29:40.705 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:40.704+0000 7f01f16d6640 1 librados: shutdown 2024-03-25T19:29:40.716 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:29:47.523 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:29:47.631 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.632+0000 7fb96371c640 10 monclient: get_monmap_and_config 2024-03-25T19:29:47.632 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.632+0000 7fb96371c640 10 monclient: build_initial_monmap 2024-03-25T19:29:47.632 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.632+0000 7fb96371c640 10 monclient: monmap: 2024-03-25T19:29:47.632 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:29:47.632 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:29:47.632 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:29:47.633263+0000 2024-03-25T19:29:47.632 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:29:47.633263+0000 2024-03-25T19:29:47.632 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:47.632 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:29:47.632 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:29:47.632 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:29:47.632 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:29:47.632 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:47.633 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.633+0000 7fb96371c640 1 Processor -- start 2024-03-25T19:29:47.634 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.633+0000 7fb96371c640 1 -- start start 2024-03-25T19:29:47.634 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.633+0000 7fb96371c640 10 monclient: init 2024-03-25T19:29:47.634 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.634+0000 7fb96371c640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:47.634 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.634+0000 7fb96371c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:29:47.634 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.634+0000 7fb96371c640 10 monclient(hunting): picked mon.noname-c con 0x7fb95c0a98f0 addr v1:172.21.15.103:6789/0 2024-03-25T19:29:47.634 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.634+0000 7fb96371c640 10 monclient(hunting): picked mon.noname-b con 0x7fb95c13f710 addr v1:172.21.15.83:6790/0 2024-03-25T19:29:47.634 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.634+0000 7fb96371c640 10 monclient(hunting): picked mon.noname-a con 0x7fb95c1430c0 addr v1:172.21.15.83:6789/0 2024-03-25T19:29:47.634 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.634+0000 7fb96371c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb95c142520 con 0x7fb95c1430c0 2024-03-25T19:29:47.634 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.634+0000 7fb96371c640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb95c0b28a0 con 0x7fb95c0a98f0 2024-03-25T19:29:47.634 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.634+0000 7fb96371c640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb95c0b2a30 con 0x7fb95c13f710 2024-03-25T19:29:47.634 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.634+0000 7fb96371c640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:47.634 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.634+0000 7fb96371c640 10 monclient(hunting): authenticate will time out at 6644.906078s 2024-03-25T19:29:47.639 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.640+0000 7fb960cb7640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fb95c13f710 0x7fb95c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:50096/0 (socket says 172.21.15.83:50096) 2024-03-25T19:29:47.640 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.640+0000 7fb960cb7640 1 -- 172.21.15.83:0/3907311752 learned_addr learned my addr 172.21.15.83:0/3907311752 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:29:47.648 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.648+0000 7fb953fff640 1 -- 172.21.15.83:0/3907311752 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1562626307 0 0) 0x7fb95c0b2a30 con 0x7fb95c13f710 2024-03-25T19:29:47.648 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.648+0000 7fb953fff640 10 monclient(hunting): _negotiate 2024-03-25T19:29:47.648 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.648+0000 7fb953fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:47.648 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.648+0000 7fb953fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:47.648 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.648+0000 7fb953fff640 10 monclient(hunting): my global_id is 4691 2024-03-25T19:29:47.648 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.648+0000 7fb953fff640 1 -- 172.21.15.83:0/3907311752 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb948003520 con 0x7fb95c13f710 2024-03-25T19:29:47.655 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.655+0000 7fb953fff640 1 -- 172.21.15.83:0/3907311752 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 675529321 0 0) 0x7fb95c142520 con 0x7fb95c1430c0 2024-03-25T19:29:47.655 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.655+0000 7fb953fff640 10 monclient(hunting): _negotiate 2024-03-25T19:29:47.655 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.655+0000 7fb953fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:47.655 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.655+0000 7fb953fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:47.655 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.655+0000 7fb953fff640 10 monclient(hunting): my global_id is 4674 2024-03-25T19:29:47.655 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.655+0000 7fb953fff640 1 -- 172.21.15.83:0/3907311752 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb95c0b2a30 con 0x7fb95c1430c0 2024-03-25T19:29:47.661 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.661+0000 7fb953fff640 1 -- 172.21.15.83:0/3907311752 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4092921360 0 0) 0x7fb948003520 con 0x7fb95c13f710 2024-03-25T19:29:47.662 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.661+0000 7fb953fff640 1 -- 172.21.15.83:0/3907311752 >> v1:172.21.15.103:6789/0 conn(0x7fb95c0a98f0 legacy=0x7fb95c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:47.662 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.661+0000 7fb953fff640 1 -- 172.21.15.83:0/3907311752 >> v1:172.21.15.83:6789/0 conn(0x7fb95c1430c0 legacy=0x7fb95c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:47.662 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.661+0000 7fb953fff640 10 monclient: _finish_hunting 0 2024-03-25T19:29:47.662 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.661+0000 7fb953fff640 1 monclient: found mon.noname-b 2024-03-25T19:29:47.662 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.661+0000 7fb953fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:47.662 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.661+0000 7fb953fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.83:6790/0 2024-03-25T19:29:47.662 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.661+0000 7fb953fff640 1 -- 172.21.15.83:0/3907311752 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb95c0b44c0 con 0x7fb95c13f710 2024-03-25T19:29:47.662 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.661+0000 7fb953fff640 10 monclient: _finish_auth 0 2024-03-25T19:29:47.662 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.661+0000 7fb953fff640 10 monclient: _check_auth_tickets 2024-03-25T19:29:47.662 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.661+0000 7fb953fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:47.662 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.661+0000 7fb953fff640 1 -- 172.21.15.83:0/3907311752 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fb94c001690 con 0x7fb95c13f710 2024-03-25T19:29:47.662 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.661+0000 7fb953fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:47.662 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.661+0000 7fb953fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:29:47.663 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.661+0000 7fb953fff640 10 monclient: dump: 2024-03-25T19:29:47.663 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:29:47.663 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:29:47.663 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:47.663 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:47.663 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:47.663 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:29:47.663 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:29:47.663 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:29:47.663 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:29:47.663 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:47.663 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.661+0000 7fb96371c640 5 monclient: authenticate success, global_id 4691 2024-03-25T19:29:47.663 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.661+0000 7fb96371c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:47.687 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.687+0000 7fb953fff640 1 -- 172.21.15.83:0/3907311752 <== mon.2 v1:172.21.15.83:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb94c0018b0 con 0x7fb95c13f710 2024-03-25T19:29:47.687 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.687+0000 7fb953fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:47.687 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.687+0000 7fb953fff640 1 -- 172.21.15.83:0/3907311752 <== mon.2 v1:172.21.15.83:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fb94c001c40 con 0x7fb95c13f710 2024-03-25T19:29:47.687 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.687+0000 7fb96371c640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:47.687 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.687+0000 7fb96371c640 4 set_mon_vals no callback set 2024-03-25T19:29:47.687 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.687+0000 7fb953fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:47.687 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.687+0000 7fb953fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:29:47.687 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.687+0000 7fb953fff640 10 monclient: dump: 2024-03-25T19:29:47.687 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:29:47.687 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:29:47.687 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:47.687 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:47.687 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:47.687 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:29:47.688 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:29:47.688 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:29:47.688 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:29:47.688 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:47.689 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.688+0000 7fb96371c640 10 monclient: shutdown 2024-03-25T19:29:47.689 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.688+0000 7fb96371c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:47.689 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.688+0000 7fb96371c640 1 -- 172.21.15.83:0/3907311752 >> v1:172.21.15.83:6790/0 conn(0x7fb95c13f710 legacy=0x7fb95c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:47.689 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.688+0000 7fb96371c640 1 -- 172.21.15.83:0/3907311752 shutdown_connections 2024-03-25T19:29:47.690 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.688+0000 7fb96371c640 1 -- 172.21.15.83:0/3907311752 >> 172.21.15.83:0/3907311752 conn(0x7fb95c0b7ea0 msgr2=0x7fb95c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:47.690 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.688+0000 7fb96371c640 1 -- 172.21.15.83:0/3907311752 shutdown_connections 2024-03-25T19:29:47.690 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.688+0000 7fb96371c640 1 -- 172.21.15.83:0/3907311752 wait complete. 2024-03-25T19:29:47.690 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.689+0000 7fb96371c640 10 monclient: build_initial_monmap 2024-03-25T19:29:47.690 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.689+0000 7fb96371c640 10 monclient: monmap: 2024-03-25T19:29:47.690 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:29:47.690 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:47.690 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:29:47.690 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:29:47.690 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:47.690 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:29:47.690 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:29:47.690 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:29:47.690 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:29:47.690 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:47.691 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.689+0000 7fb96371c640 1 librados: starting msgr at 2024-03-25T19:29:47.691 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.689+0000 7fb96371c640 1 librados: starting objecter 2024-03-25T19:29:47.691 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.690+0000 7fb96371c640 1 Processor -- start 2024-03-25T19:29:47.691 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.690+0000 7fb96371c640 1 -- start start 2024-03-25T19:29:47.691 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.690+0000 7fb96371c640 1 librados: setting wanted keys 2024-03-25T19:29:47.691 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.690+0000 7fb96371c640 1 librados: calling monclient init 2024-03-25T19:29:47.691 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.690+0000 7fb96371c640 10 monclient: init 2024-03-25T19:29:47.692 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.690+0000 7fb96371c640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:47.692 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.690+0000 7fb96371c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:29:47.692 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.690+0000 7fb96371c640 10 monclient(hunting): picked mon.noname-b con 0x7fb95c0a98f0 addr v1:172.21.15.103:6789/0 2024-03-25T19:29:47.692 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.690+0000 7fb96371c640 10 monclient(hunting): picked mon.noname-c con 0x7fb95c13f710 addr v1:172.21.15.83:6790/0 2024-03-25T19:29:47.692 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.691+0000 7fb96371c640 10 monclient(hunting): picked mon.noname-a con 0x7fb95c1430c0 addr v1:172.21.15.83:6789/0 2024-03-25T19:29:47.692 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.691+0000 7fb96371c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb95c0aba00 con 0x7fb95c1430c0 2024-03-25T19:29:47.692 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.691+0000 7fb96371c640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb95c1d7490 con 0x7fb95c0a98f0 2024-03-25T19:29:47.693 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.691+0000 7fb96371c640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb95c1d8360 con 0x7fb95c13f710 2024-03-25T19:29:47.693 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.691+0000 7fb96371c640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:47.693 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.691+0000 7fb96371c640 10 monclient(hunting): authenticate will time out at 6644.962936s 2024-03-25T19:29:47.698 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.698+0000 7fb9614b8640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fb95c0a98f0 0x7fb95c1386e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:49292/0 (socket says 172.21.15.83:49292) 2024-03-25T19:29:47.698 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.698+0000 7fb9614b8640 1 -- 172.21.15.83:0/1421757759 learned_addr learned my addr 172.21.15.83:0/1421757759 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:29:47.699 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.700+0000 7fb951ffb640 10 client.?.objecter ms_handle_connect 0x7fb95c0a98f0 2024-03-25T19:29:47.699 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.700+0000 7fb951ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:47.700 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.700+0000 7fb951ffb640 10 client.?.objecter ms_handle_connect 0x7fb95c13f710 2024-03-25T19:29:47.700 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.700+0000 7fb951ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:47.713 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.714+0000 7fb951ffb640 10 client.?.objecter ms_handle_connect 0x7fb95c1430c0 2024-03-25T19:29:47.713 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.714+0000 7fb951ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:47.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.715+0000 7fb951ffb640 1 -- 172.21.15.83:0/1421757759 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 510265131 0 0) 0x7fb95c1d7490 con 0x7fb95c0a98f0 2024-03-25T19:29:47.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.715+0000 7fb951ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:29:47.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.715+0000 7fb951ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:47.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.715+0000 7fb951ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:47.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.715+0000 7fb951ffb640 10 monclient(hunting): my global_id is 4693 2024-03-25T19:29:47.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.715+0000 7fb951ffb640 1 -- 172.21.15.83:0/1421757759 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb940004eb0 con 0x7fb95c0a98f0 2024-03-25T19:29:47.716 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.716+0000 7fb951ffb640 1 -- 172.21.15.83:0/1421757759 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1418938438 0 0) 0x7fb95c1d8360 con 0x7fb95c13f710 2024-03-25T19:29:47.716 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.716+0000 7fb951ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:29:47.716 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.716+0000 7fb951ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:47.716 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.716+0000 7fb951ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:47.716 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.716+0000 7fb951ffb640 10 monclient(hunting): my global_id is 4694 2024-03-25T19:29:47.716 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.716+0000 7fb951ffb640 1 -- 172.21.15.83:0/1421757759 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb95c1d7490 con 0x7fb95c13f710 2024-03-25T19:29:47.727 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.728+0000 7fb951ffb640 1 -- 172.21.15.83:0/1421757759 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3442901032 0 0) 0x7fb95c0aba00 con 0x7fb95c1430c0 2024-03-25T19:29:47.727 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.728+0000 7fb951ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:29:47.727 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.728+0000 7fb951ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:47.727 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.728+0000 7fb951ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:47.727 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.728+0000 7fb951ffb640 10 monclient(hunting): my global_id is 4677 2024-03-25T19:29:47.727 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.728+0000 7fb951ffb640 1 -- 172.21.15.83:0/1421757759 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb95c1d8360 con 0x7fb95c1430c0 2024-03-25T19:29:47.731 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.731+0000 7fb951ffb640 1 -- 172.21.15.83:0/1421757759 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3737939712 0 0) 0x7fb940004eb0 con 0x7fb95c0a98f0 2024-03-25T19:29:47.731 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.731+0000 7fb951ffb640 1 -- 172.21.15.83:0/1421757759 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb95c0aba00 con 0x7fb95c0a98f0 2024-03-25T19:29:47.731 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.731+0000 7fb951ffb640 1 -- 172.21.15.83:0/1421757759 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fb958001a30 con 0x7fb95c0a98f0 2024-03-25T19:29:47.731 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.731+0000 7fb951ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:47.731 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.731+0000 7fb951ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:29:47.731 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.731+0000 7fb951ffb640 10 monclient(hunting): dump: 2024-03-25T19:29:47.731 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:29:47.731 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:29:47.731 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:47.731 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:47.731 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:47.731 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:29:47.731 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:29:47.731 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:29:47.731 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:29:47.732 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:47.732 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.731+0000 7fb951ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:47.733 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.733+0000 7fb951ffb640 1 -- 172.21.15.83:0/1421757759 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2897647192 0 0) 0x7fb95c1d7490 con 0x7fb95c13f710 2024-03-25T19:29:47.733 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.733+0000 7fb951ffb640 1 -- 172.21.15.83:0/1421757759 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb940004eb0 con 0x7fb95c13f710 2024-03-25T19:29:47.733 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.733+0000 7fb951ffb640 1 -- 172.21.15.83:0/1421757759 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fb94c0015d0 con 0x7fb95c13f710 2024-03-25T19:29:47.733 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.733+0000 7fb951ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:47.733 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.733+0000 7fb951ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:29:47.733 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.733+0000 7fb951ffb640 10 monclient(hunting): dump: 2024-03-25T19:29:47.733 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:29:47.733 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:29:47.733 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:47.733 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:47.733 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:47.734 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:29:47.734 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:29:47.734 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:29:47.734 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:29:47.734 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:47.742 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.742+0000 7fb951ffb640 1 -- 172.21.15.83:0/1421757759 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3641856038 0 0) 0x7fb95c1d8360 con 0x7fb95c1430c0 2024-03-25T19:29:47.742 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.742+0000 7fb951ffb640 1 -- 172.21.15.83:0/1421757759 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb95c1d7490 con 0x7fb95c1430c0 2024-03-25T19:29:47.742 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.742+0000 7fb951ffb640 1 -- 172.21.15.83:0/1421757759 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fb954001b90 con 0x7fb95c1430c0 2024-03-25T19:29:47.742 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.742+0000 7fb951ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:47.742 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.742+0000 7fb951ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:29:47.742 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.742+0000 7fb951ffb640 10 monclient(hunting): dump: 2024-03-25T19:29:47.742 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:29:47.742 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:29:47.742 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:47.742 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:47.743 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:47.743 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:29:47.743 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:29:47.743 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:29:47.743 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:29:47.743 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:47.744 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.744+0000 7fb951ffb640 1 -- 172.21.15.83:0/1421757759 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2608209745 0 0) 0x7fb940004eb0 con 0x7fb95c13f710 2024-03-25T19:29:47.744 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.744+0000 7fb951ffb640 1 -- 172.21.15.83:0/1421757759 >> v1:172.21.15.103:6789/0 conn(0x7fb95c0a98f0 legacy=0x7fb95c1386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:47.744 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.744+0000 7fb951ffb640 1 -- 172.21.15.83:0/1421757759 >> v1:172.21.15.83:6789/0 conn(0x7fb95c1430c0 legacy=0x7fb95c133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:47.744 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.744+0000 7fb951ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:29:47.744 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.744+0000 7fb951ffb640 1 monclient: found mon.c 2024-03-25T19:29:47.744 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.744+0000 7fb951ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:47.744 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.744+0000 7fb951ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:29:47.744 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.744+0000 7fb951ffb640 1 -- 172.21.15.83:0/1421757759 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb95c0abc30 con 0x7fb95c13f710 2024-03-25T19:29:47.744 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.744+0000 7fb951ffb640 10 monclient: _finish_auth 0 2024-03-25T19:29:47.744 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.744+0000 7fb951ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:29:47.744 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.744+0000 7fb951ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:47.745 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.744+0000 7fb96371c640 5 monclient: authenticate success, global_id 4694 2024-03-25T19:29:47.745 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.744+0000 7fb96371c640 10 monclient: _renew_subs 2024-03-25T19:29:47.745 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.744+0000 7fb96371c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:29:47.745 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.744+0000 7fb96371c640 1 -- 172.21.15.83:0/1421757759 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb95c137940 con 0x7fb95c13f710 2024-03-25T19:29:47.745 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.744+0000 7fb96371c640 10 client.4694.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:47.745 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.744+0000 7fb96371c640 10 monclient: _renew_subs 2024-03-25T19:29:47.745 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.744+0000 7fb96371c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:29:47.745 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.744+0000 7fb96371c640 1 -- 172.21.15.83:0/1421757759 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb95c135220 con 0x7fb95c13f710 2024-03-25T19:29:47.745 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.744+0000 7fb96371c640 1 librados: init done 2024-03-25T19:29:47.745 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.745+0000 7fb96371c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:47.746 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.745+0000 7fb96371c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:47.746 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.745+0000 7fb96371c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:29:47.746 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:47.745+0000 7fb96371c640 1 -- 172.21.15.83:0/1421757759 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb95c0627b0 con 0x7fb95c13f710 2024-03-25T19:29:48.086 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:48.086+0000 7fb951ffb640 1 -- 172.21.15.83:0/1421757759 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb94c001730 con 0x7fb95c13f710 2024-03-25T19:29:48.086 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:48.086+0000 7fb951ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:48.086 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:48.086+0000 7fb951ffb640 1 -- 172.21.15.83:0/1421757759 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fb94c004c30 con 0x7fb95c13f710 2024-03-25T19:29:48.086 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:48.086+0000 7fb951ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:48.087 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:48.086+0000 7fb951ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:29:48.087 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:48.086+0000 7fb951ffb640 10 monclient: dump: 2024-03-25T19:29:48.087 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:29:48.087 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:29:48.087 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:48.087 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:48.087 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:48.087 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:29:48.087 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:29:48.087 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:29:48.087 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:29:48.087 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:48.087 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:48.086+0000 7fb9537fe640 4 set_mon_vals no callback set 2024-03-25T19:29:48.088 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:48.087+0000 7fb951ffb640 1 -- 172.21.15.83:0/1421757759 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7fb94c01ac00 con 0x7fb95c13f710 2024-03-25T19:29:48.088 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:48.088+0000 7fb9614b8640 10 monclient: get_auth_request con 0x7fb94006e150 auth_method 0 2024-03-25T19:29:48.088 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:48.088+0000 7fb951ffb640 10 client.4694.objecter ms_handle_connect 0x7fb94006e150 2024-03-25T19:29:48.092 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:48.093+0000 7fb951ffb640 1 -- 172.21.15.83:0/1421757759 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 816112689 0 0) 0x7fb94c0547f0 con 0x7fb95c13f710 2024-03-25T19:29:48.092 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:48.093+0000 7fb951ffb640 10 client.4694.objecter ms_dispatch 0x7fb95c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:29:48.093 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:48.093+0000 7fb951ffb640 3 client.4694.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:29:48.093 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:48.093+0000 7fb951ffb640 3 client.4694.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:29:48.093 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:48.093+0000 7fb951ffb640 20 client.4694.objecter dump_active .. 0 homeless 2024-03-25T19:29:48.093 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:48.093+0000 7fb951ffb640 1 -- 172.21.15.83:0/1421757759 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb94c050130 con 0x7fb95c13f710 2024-03-25T19:29:48.093 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:48.093+0000 7fb951ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:48.093 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:48.093+0000 7fb951ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:48.343 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:48.343+0000 7fb96371c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:48.343 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:48.343+0000 7fb96371c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:48.343 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:48.343+0000 7fb96371c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:29:48.343 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:48.343+0000 7fb96371c640 1 -- 172.21.15.83:0/1421757759 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb95c069f40 con 0x7fb95c13f710 2024-03-25T19:29:48.367 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:48.367+0000 7fb951ffb640 1 -- 172.21.15.83:0/1421757759 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2915350783) 0x7fb94c052230 con 0x7fb95c13f710 2024-03-25T19:29:48.367 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:48.368+0000 7fb951ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:48.367 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:48.368+0000 7fb951ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:48.367 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:29:48.368 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:19:50.378282+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:19:50.378282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:29:48.370 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:48.370+0000 7fb96371c640 10 librados: watch_flush enter 2024-03-25T19:29:48.370 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:48.370+0000 7fb96371c640 10 librados: watch_flush exit 2024-03-25T19:29:48.370 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:48.370+0000 7fb96371c640 20 client.4694.objecter shutdown clearing up homeless session... 2024-03-25T19:29:48.370 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:48.370+0000 7fb96371c640 10 client.4694.objecter successfully canceled tick 2024-03-25T19:29:48.370 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:48.370+0000 7fb96371c640 1 -- 172.21.15.83:0/1421757759 >> v1:172.21.15.103:6800/345072327 conn(0x7fb94006e150 legacy=0x7fb940070610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:48.370 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:48.370+0000 7fb96371c640 10 monclient: shutdown 2024-03-25T19:29:48.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:48.370+0000 7fb96371c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:48.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:48.370+0000 7fb96371c640 1 -- 172.21.15.83:0/1421757759 >> v1:172.21.15.83:6790/0 conn(0x7fb95c13f710 legacy=0x7fb95c1d7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:48.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:48.370+0000 7fb96371c640 1 -- 172.21.15.83:0/1421757759 shutdown_connections 2024-03-25T19:29:48.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:48.370+0000 7fb96371c640 1 -- 172.21.15.83:0/1421757759 >> 172.21.15.83:0/1421757759 conn(0x7fb95c0b7ea0 msgr2=0x7fb95c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:48.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:48.370+0000 7fb96371c640 1 -- 172.21.15.83:0/1421757759 shutdown_connections 2024-03-25T19:29:48.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:48.370+0000 7fb96371c640 1 -- 172.21.15.83:0/1421757759 wait complete. 2024-03-25T19:29:48.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:48.370+0000 7fb96371c640 1 librados: shutdown 2024-03-25T19:29:48.382 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:29:49.265 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:10:31.192 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:49.268 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:10:31.196 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:49.295 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:10:31.223 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:49.336 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:10:31.246 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:49.347 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:10:31.274 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:49.347 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:10:31.257 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:49.354 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:10:31.282 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:49.372 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:10:31.283 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:49.386 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:10:31.313 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:55.189 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:29:55.289 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.289+0000 7f96b2db6640 10 monclient: get_monmap_and_config 2024-03-25T19:29:55.289 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.289+0000 7f96b2db6640 10 monclient: build_initial_monmap 2024-03-25T19:29:55.289 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.289+0000 7f96b2db6640 10 monclient: monmap: 2024-03-25T19:29:55.289 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:29:55.289 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:29:55.289 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:29:55.290583+0000 2024-03-25T19:29:55.290 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:29:55.290583+0000 2024-03-25T19:29:55.290 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:55.290 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:29:55.290 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:29:55.290 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:29:55.290 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:29:55.290 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:55.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.290+0000 7f96b2db6640 1 Processor -- start 2024-03-25T19:29:55.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.290+0000 7f96b2db6640 1 -- start start 2024-03-25T19:29:55.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.290+0000 7f96b2db6640 10 monclient: init 2024-03-25T19:29:55.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.291+0000 7f96b2db6640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:55.292 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.291+0000 7f96b2db6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:29:55.292 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.291+0000 7f96b2db6640 10 monclient(hunting): picked mon.noname-b con 0x7f96ac0b4d60 addr v1:172.21.15.83:6790/0 2024-03-25T19:29:55.292 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.291+0000 7f96b2db6640 10 monclient(hunting): picked mon.noname-c con 0x7f96ac0b1730 addr v1:172.21.15.103:6789/0 2024-03-25T19:29:55.292 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.291+0000 7f96b2db6640 10 monclient(hunting): picked mon.noname-a con 0x7f96ac14be90 addr v1:172.21.15.83:6789/0 2024-03-25T19:29:55.292 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.291+0000 7f96b2db6640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96ac0af940 con 0x7f96ac14be90 2024-03-25T19:29:55.292 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.291+0000 7f96b2db6640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96ac0b0a70 con 0x7f96ac0b1730 2024-03-25T19:29:55.292 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.291+0000 7f96b2db6640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96ac0b0d50 con 0x7f96ac0b4d60 2024-03-25T19:29:55.292 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.291+0000 7f96b2db6640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:55.292 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.291+0000 7f96b2db6640 10 monclient(hunting): authenticate will time out at 6652.563404s 2024-03-25T19:29:55.293 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.292+0000 7f96abfff640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f96ac0b1730 0x7f96ac0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:53916/0 (socket says 172.21.15.83:53916) 2024-03-25T19:29:55.293 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.292+0000 7f96abfff640 1 -- 172.21.15.83:0/1795802204 learned_addr learned my addr 172.21.15.83:0/1795802204 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:29:55.312 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.312+0000 7f96ab7fe640 1 -- 172.21.15.83:0/1795802204 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1225775570 0 0) 0x7f96ac0af940 con 0x7f96ac14be90 2024-03-25T19:29:55.312 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.312+0000 7f96ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:29:55.312 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.312+0000 7f96ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:55.312 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.312+0000 7f96ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:55.312 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.312+0000 7f96ab7fe640 10 monclient(hunting): my global_id is 4680 2024-03-25T19:29:55.312 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.312+0000 7f96ab7fe640 1 -- 172.21.15.83:0/1795802204 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9694003520 con 0x7f96ac14be90 2024-03-25T19:29:55.318 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.318+0000 7f96ab7fe640 1 -- 172.21.15.83:0/1795802204 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3116795113 0 0) 0x7f9694003520 con 0x7f96ac14be90 2024-03-25T19:29:55.318 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.318+0000 7f96ab7fe640 1 -- 172.21.15.83:0/1795802204 >> v1:172.21.15.83:6790/0 conn(0x7f96ac0b4d60 legacy=0x7f96ac13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:29:55.318 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.318+0000 7f96ab7fe640 1 -- 172.21.15.83:0/1795802204 >> v1:172.21.15.103:6789/0 conn(0x7f96ac0b1730 legacy=0x7f96ac0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:55.319 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.318+0000 7f96ab7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:29:55.319 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.318+0000 7f96ab7fe640 1 monclient: found mon.noname-a 2024-03-25T19:29:55.319 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.318+0000 7f96ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:55.319 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.318+0000 7f96ab7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.83:6789/0 2024-03-25T19:29:55.319 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.318+0000 7f96ab7fe640 1 -- 172.21.15.83:0/1795802204 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96ac0b1030 con 0x7f96ac14be90 2024-03-25T19:29:55.319 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.318+0000 7f96ab7fe640 10 monclient: _finish_auth 0 2024-03-25T19:29:55.319 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.318+0000 7f96ab7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:29:55.319 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.318+0000 7f96ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:55.319 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.318+0000 7f96ab7fe640 1 -- 172.21.15.83:0/1795802204 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f96a4001a10 con 0x7f96ac14be90 2024-03-25T19:29:55.319 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.318+0000 7f96ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:55.319 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.318+0000 7f96ab7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:29:55.319 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.318+0000 7f96ab7fe640 10 monclient: dump: 2024-03-25T19:29:55.319 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:29:55.319 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:29:55.319 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:55.320 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:55.320 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:55.320 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:29:55.320 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:29:55.320 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:29:55.320 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:29:55.320 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:55.320 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.318+0000 7f96b2db6640 5 monclient: authenticate success, global_id 4680 2024-03-25T19:29:55.320 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.318+0000 7f96b2db6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:55.327 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.327+0000 7f96ab7fe640 1 -- 172.21.15.83:0/1795802204 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96a4001be0 con 0x7f96ac14be90 2024-03-25T19:29:55.327 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.327+0000 7f96ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:55.327 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.327+0000 7f96ab7fe640 1 -- 172.21.15.83:0/1795802204 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f96a4001ff0 con 0x7f96ac14be90 2024-03-25T19:29:55.328 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.327+0000 7f96ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:55.328 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.327+0000 7f96ab7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:29:55.328 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.327+0000 7f96ab7fe640 10 monclient: dump: 2024-03-25T19:29:55.328 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:29:55.328 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:29:55.328 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:55.328 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:55.328 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:55.328 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:29:55.328 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:29:55.328 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:29:55.328 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:29:55.328 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:55.328 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.327+0000 7f96b2db6640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:55.329 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.327+0000 7f96b2db6640 4 set_mon_vals no callback set 2024-03-25T19:29:55.331 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.328+0000 7f96b2db6640 10 monclient: shutdown 2024-03-25T19:29:55.331 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.328+0000 7f96b2db6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:55.331 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.328+0000 7f96b2db6640 1 -- 172.21.15.83:0/1795802204 >> v1:172.21.15.83:6789/0 conn(0x7f96ac14be90 legacy=0x7f96ac14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:55.331 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.328+0000 7f96b2db6640 1 -- 172.21.15.83:0/1795802204 shutdown_connections 2024-03-25T19:29:55.331 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.329+0000 7f96b2db6640 1 -- 172.21.15.83:0/1795802204 >> 172.21.15.83:0/1795802204 conn(0x7f96ac0617a0 msgr2=0x7f96ac061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:55.331 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.329+0000 7f96b2db6640 1 -- 172.21.15.83:0/1795802204 shutdown_connections 2024-03-25T19:29:55.331 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.329+0000 7f96b2db6640 1 -- 172.21.15.83:0/1795802204 wait complete. 2024-03-25T19:29:55.331 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.329+0000 7f96b2db6640 10 monclient: build_initial_monmap 2024-03-25T19:29:55.332 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.329+0000 7f96b2db6640 10 monclient: monmap: 2024-03-25T19:29:55.332 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:29:55.332 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:55.332 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:29:55.332 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:29:55.332 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:55.332 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:29:55.332 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:29:55.332 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:29:55.332 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:29:55.332 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:55.332 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.330+0000 7f96b2db6640 1 librados: starting msgr at 2024-03-25T19:29:55.332 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.330+0000 7f96b2db6640 1 librados: starting objecter 2024-03-25T19:29:55.332 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.330+0000 7f96b2db6640 1 Processor -- start 2024-03-25T19:29:55.332 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.330+0000 7f96b2db6640 1 -- start start 2024-03-25T19:29:55.332 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.330+0000 7f96b2db6640 1 librados: setting wanted keys 2024-03-25T19:29:55.333 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.330+0000 7f96b2db6640 1 librados: calling monclient init 2024-03-25T19:29:55.333 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.330+0000 7f96b2db6640 10 monclient: init 2024-03-25T19:29:55.333 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.331+0000 7f96b2db6640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:55.333 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.331+0000 7f96b2db6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:29:55.333 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.331+0000 7f96b2db6640 10 monclient(hunting): picked mon.noname-b con 0x7f96ac0b1730 addr v1:172.21.15.103:6789/0 2024-03-25T19:29:55.333 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.331+0000 7f96b2db6640 10 monclient(hunting): picked mon.noname-c con 0x7f96ac0b4d60 addr v1:172.21.15.83:6790/0 2024-03-25T19:29:55.333 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.331+0000 7f96b2db6640 10 monclient(hunting): picked mon.noname-a con 0x7f96ac14be90 addr v1:172.21.15.83:6789/0 2024-03-25T19:29:55.333 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.331+0000 7f96b2db6640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96ac1daf60 con 0x7f96ac14be90 2024-03-25T19:29:55.333 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.331+0000 7f96b2db6640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96ac1422d0 con 0x7f96ac0b1730 2024-03-25T19:29:55.333 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.331+0000 7f96b2db6640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96ac140a90 con 0x7f96ac0b4d60 2024-03-25T19:29:55.333 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.331+0000 7f96b2db6640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:55.333 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.331+0000 7f96b2db6640 10 monclient(hunting): authenticate will time out at 6652.603263s 2024-03-25T19:29:55.353 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.354+0000 7f96abfff640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f96ac0b4d60 0x7f96ac1db3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:54680/0 (socket says 172.21.15.83:54680) 2024-03-25T19:29:55.353 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.354+0000 7f96abfff640 1 -- 172.21.15.83:0/3618842428 learned_addr learned my addr 172.21.15.83:0/3618842428 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:29:55.355 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.355+0000 7f96a97fa640 10 client.?.objecter ms_handle_connect 0x7f96ac0b4d60 2024-03-25T19:29:55.355 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.355+0000 7f96a97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:55.357 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.357+0000 7f96a97fa640 10 client.?.objecter ms_handle_connect 0x7f96ac14be90 2024-03-25T19:29:55.357 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.357+0000 7f96a97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:55.365 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.365+0000 7f96a97fa640 1 -- 172.21.15.83:0/3618842428 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 445697374 0 0) 0x7f96ac1daf60 con 0x7f96ac14be90 2024-03-25T19:29:55.365 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.365+0000 7f96a97fa640 10 monclient(hunting): _negotiate 2024-03-25T19:29:55.365 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.365+0000 7f96a97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:55.365 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.365+0000 7f96a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:55.365 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.365+0000 7f96a97fa640 10 monclient(hunting): my global_id is 4683 2024-03-25T19:29:55.365 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.365+0000 7f96a97fa640 1 -- 172.21.15.83:0/3618842428 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9698004eb0 con 0x7f96ac14be90 2024-03-25T19:29:55.366 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.367+0000 7f96a97fa640 10 client.?.objecter ms_handle_connect 0x7f96ac0b1730 2024-03-25T19:29:55.367 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.367+0000 7f96a97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:55.368 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.368+0000 7f96a97fa640 1 -- 172.21.15.83:0/3618842428 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3101292302 0 0) 0x7f96ac140a90 con 0x7f96ac0b4d60 2024-03-25T19:29:55.368 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.368+0000 7f96a97fa640 10 monclient(hunting): _negotiate 2024-03-25T19:29:55.368 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.368+0000 7f96a97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:55.368 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.368+0000 7f96a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:55.368 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.368+0000 7f96a97fa640 10 monclient(hunting): my global_id is 4697 2024-03-25T19:29:55.368 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.368+0000 7f96a97fa640 1 -- 172.21.15.83:0/3618842428 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96ac1daf60 con 0x7f96ac0b4d60 2024-03-25T19:29:55.382 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.382+0000 7f96a97fa640 1 -- 172.21.15.83:0/3618842428 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 495242870 0 0) 0x7f96ac1422d0 con 0x7f96ac0b1730 2024-03-25T19:29:55.382 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.382+0000 7f96a97fa640 10 monclient(hunting): _negotiate 2024-03-25T19:29:55.382 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.382+0000 7f96a97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:55.382 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.382+0000 7f96a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:55.382 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.382+0000 7f96a97fa640 10 monclient(hunting): my global_id is 4699 2024-03-25T19:29:55.382 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.382+0000 7f96a97fa640 1 -- 172.21.15.83:0/3618842428 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96ac140a90 con 0x7f96ac0b1730 2024-03-25T19:29:55.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.383+0000 7f96a97fa640 1 -- 172.21.15.83:0/3618842428 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 175625228 0 0) 0x7f96ac1daf60 con 0x7f96ac0b4d60 2024-03-25T19:29:55.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.383+0000 7f96a97fa640 1 -- 172.21.15.83:0/3618842428 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96ac1422d0 con 0x7f96ac0b4d60 2024-03-25T19:29:55.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.383+0000 7f96a97fa640 1 -- 172.21.15.83:0/3618842428 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f96a00018f0 con 0x7f96ac0b4d60 2024-03-25T19:29:55.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.383+0000 7f96a97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:55.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.383+0000 7f96a97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:29:55.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.383+0000 7f96a97fa640 10 monclient(hunting): dump: 2024-03-25T19:29:55.384 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:29:55.384 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:29:55.384 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:55.384 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:55.384 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:55.384 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:29:55.384 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:29:55.384 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:29:55.384 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:29:55.384 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:55.384 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.383+0000 7f96a97fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:55.399 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.398+0000 7f96a97fa640 1 -- 172.21.15.83:0/3618842428 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2783234362 0 0) 0x7f96ac1422d0 con 0x7f96ac0b4d60 2024-03-25T19:29:55.399 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.398+0000 7f96a97fa640 1 -- 172.21.15.83:0/3618842428 >> v1:172.21.15.103:6789/0 conn(0x7f96ac0b1730 legacy=0x7f96ac140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:55.399 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.398+0000 7f96a97fa640 1 -- 172.21.15.83:0/3618842428 >> v1:172.21.15.83:6789/0 conn(0x7f96ac14be90 legacy=0x7f96ac1d7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:55.399 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.398+0000 7f96a97fa640 10 monclient: _finish_hunting 0 2024-03-25T19:29:55.399 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.398+0000 7f96a97fa640 1 monclient: found mon.c 2024-03-25T19:29:55.399 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.398+0000 7f96a97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:55.399 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.398+0000 7f96a97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:29:55.399 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.398+0000 7f96a97fa640 1 -- 172.21.15.83:0/3618842428 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96ac1d79e0 con 0x7f96ac0b4d60 2024-03-25T19:29:55.399 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.398+0000 7f96a97fa640 10 monclient: _finish_auth 0 2024-03-25T19:29:55.399 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.398+0000 7f96a97fa640 10 monclient: _check_auth_tickets 2024-03-25T19:29:55.399 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.398+0000 7f96a97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:55.399 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.398+0000 7f96b2db6640 5 monclient: authenticate success, global_id 4697 2024-03-25T19:29:55.399 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.398+0000 7f96b2db6640 10 monclient: _renew_subs 2024-03-25T19:29:55.400 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.398+0000 7f96b2db6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:29:55.400 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.398+0000 7f96b2db6640 1 -- 172.21.15.83:0/3618842428 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96ac1dbb00 con 0x7f96ac0b4d60 2024-03-25T19:29:55.400 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.398+0000 7f96b2db6640 10 client.4697.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:55.400 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.398+0000 7f96b2db6640 10 monclient: _renew_subs 2024-03-25T19:29:55.400 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.398+0000 7f96b2db6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:29:55.400 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.398+0000 7f96b2db6640 1 -- 172.21.15.83:0/3618842428 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f96ac1d87a0 con 0x7f96ac0b4d60 2024-03-25T19:29:55.400 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.398+0000 7f96b2db6640 1 librados: init done 2024-03-25T19:29:55.401 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.400+0000 7f96b2db6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:55.401 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.400+0000 7f96b2db6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:55.401 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.400+0000 7f96b2db6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:29:55.401 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.400+0000 7f96b2db6640 1 -- 172.21.15.83:0/3618842428 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96ac140c20 con 0x7f96ac0b4d60 2024-03-25T19:29:55.710 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.710+0000 7f96a97fa640 1 -- 172.21.15.83:0/3618842428 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96a0001d20 con 0x7f96ac0b4d60 2024-03-25T19:29:55.710 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.710+0000 7f96a97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:55.710 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.710+0000 7f96a97fa640 1 -- 172.21.15.83:0/3618842428 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f96a0004a40 con 0x7f96ac0b4d60 2024-03-25T19:29:55.710 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.710+0000 7f96a97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:55.710 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.710+0000 7f96a97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:29:55.710 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.710+0000 7f96a97fa640 10 monclient: dump: 2024-03-25T19:29:55.710 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:29:55.710 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:29:55.711 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:55.711 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:29:55.711 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:55.711 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:29:55.711 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:29:55.711 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:29:55.711 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:29:55.711 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:29:55.711 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.710+0000 7f96aaffd640 4 set_mon_vals no callback set 2024-03-25T19:29:55.712 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.711+0000 7f96a97fa640 1 -- 172.21.15.83:0/3618842428 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7f96a0004c60 con 0x7f96ac0b4d60 2024-03-25T19:29:55.712 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.711+0000 7f96b0b52640 10 monclient: get_auth_request con 0x7f969806de20 auth_method 0 2024-03-25T19:29:55.712 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.712+0000 7f96a97fa640 10 client.4697.objecter ms_handle_connect 0x7f969806de20 2024-03-25T19:29:55.717 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.717+0000 7f96a97fa640 1 -- 172.21.15.83:0/3618842428 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2659774754 0 0) 0x7f96a00547c0 con 0x7f96ac0b4d60 2024-03-25T19:29:55.718 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.717+0000 7f96a97fa640 10 client.4697.objecter ms_dispatch 0x7f96ac000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:29:55.718 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.717+0000 7f96a97fa640 3 client.4697.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:29:55.718 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.717+0000 7f96a97fa640 3 client.4697.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:29:55.718 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.717+0000 7f96a97fa640 20 client.4697.objecter dump_active .. 0 homeless 2024-03-25T19:29:55.718 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.717+0000 7f96a97fa640 1 -- 172.21.15.83:0/3618842428 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f96a0050100 con 0x7f96ac0b4d60 2024-03-25T19:29:55.718 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.717+0000 7f96a97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:55.718 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.717+0000 7f96a97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:55.969 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.969+0000 7f96b2db6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:55.969 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.969+0000 7f96b2db6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:55.969 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.969+0000 7f96b2db6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:29:55.969 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.969+0000 7f96b2db6640 1 -- 172.21.15.83:0/3618842428 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f96ac1dbe50 con 0x7f96ac0b4d60 2024-03-25T19:29:55.997 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.997+0000 7f96a97fa640 1 -- 172.21.15.83:0/3618842428 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1070525753) 0x7f96a0052200 con 0x7f96ac0b4d60 2024-03-25T19:29:55.997 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.997+0000 7f96a97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:55.997 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:55.997+0000 7f96a97fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:55.997 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:29:55.998 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":11,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:29:48.933916+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:48.933916+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:29:56.002 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:56.000+0000 7f96b2db6640 10 librados: watch_flush enter 2024-03-25T19:29:56.002 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:56.000+0000 7f96b2db6640 10 librados: watch_flush exit 2024-03-25T19:29:56.002 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:56.000+0000 7f96b2db6640 20 client.4697.objecter shutdown clearing up homeless session... 2024-03-25T19:29:56.003 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:56.000+0000 7f96b2db6640 10 client.4697.objecter successfully canceled tick 2024-03-25T19:29:56.003 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:56.000+0000 7f96b2db6640 1 -- 172.21.15.83:0/3618842428 >> v1:172.21.15.103:6800/345072327 conn(0x7f969806de20 legacy=0x7f96980702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:56.003 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:56.000+0000 7f96b2db6640 10 monclient: shutdown 2024-03-25T19:29:56.003 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:56.000+0000 7f96b2db6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:56.003 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:56.000+0000 7f96b2db6640 1 -- 172.21.15.83:0/3618842428 >> v1:172.21.15.83:6790/0 conn(0x7f96ac0b4d60 legacy=0x7f96ac1db3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:56.003 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:56.000+0000 7f96b2db6640 1 -- 172.21.15.83:0/3618842428 shutdown_connections 2024-03-25T19:29:56.003 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:56.000+0000 7f96b2db6640 1 -- 172.21.15.83:0/3618842428 >> 172.21.15.83:0/3618842428 conn(0x7f96ac0617a0 msgr2=0x7f96ac145130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:56.003 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:56.000+0000 7f96b2db6640 1 -- 172.21.15.83:0/3618842428 shutdown_connections 2024-03-25T19:29:56.003 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:56.000+0000 7f96b2db6640 1 -- 172.21.15.83:0/3618842428 wait complete. 2024-03-25T19:29:56.003 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:29:56.000+0000 7f96b2db6640 1 librados: shutdown 2024-03-25T19:29:56.013 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:02.818 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:02.937 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.937+0000 7f64b437b640 10 monclient: get_monmap_and_config 2024-03-25T19:30:02.937 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.937+0000 7f64b437b640 10 monclient: build_initial_monmap 2024-03-25T19:30:02.937 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.937+0000 7f64b437b640 10 monclient: monmap: 2024-03-25T19:30:02.937 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:30:02.937 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:30:02.937 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:30:02.938295+0000 2024-03-25T19:30:02.937 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:30:02.938295+0000 2024-03-25T19:30:02.937 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:02.937 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:30:02.937 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:30:02.937 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:30:02.938 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:30:02.938 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:02.939 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.938+0000 7f64b437b640 1 Processor -- start 2024-03-25T19:30:02.939 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.938+0000 7f64b437b640 1 -- start start 2024-03-25T19:30:02.939 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.938+0000 7f64b437b640 10 monclient: init 2024-03-25T19:30:02.939 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.938+0000 7f64b437b640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:02.939 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.938+0000 7f64b437b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:30:02.939 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.939+0000 7f64b437b640 10 monclient(hunting): picked mon.noname-b con 0x7f64ac0a98f0 addr v1:172.21.15.83:6790/0 2024-03-25T19:30:02.939 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.939+0000 7f64b437b640 10 monclient(hunting): picked mon.noname-c con 0x7f64ac13f710 addr v1:172.21.15.103:6789/0 2024-03-25T19:30:02.939 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.939+0000 7f64b437b640 10 monclient(hunting): picked mon.noname-a con 0x7f64ac1430c0 addr v1:172.21.15.83:6789/0 2024-03-25T19:30:02.939 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.939+0000 7f64b437b640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64ac142520 con 0x7f64ac1430c0 2024-03-25T19:30:02.939 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.939+0000 7f64b437b640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64ac0b28a0 con 0x7f64ac13f710 2024-03-25T19:30:02.939 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.939+0000 7f64b437b640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64ac0b2a30 con 0x7f64ac0a98f0 2024-03-25T19:30:02.939 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.939+0000 7f64b437b640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:02.940 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.939+0000 7f64b437b640 10 monclient(hunting): authenticate will time out at 6660.210911s 2024-03-25T19:30:02.944 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.944+0000 7f64b2918640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f64ac1430c0 0x7f64ac14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:58196/0 (socket says 172.21.15.83:58196) 2024-03-25T19:30:02.944 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.944+0000 7f64b2918640 1 -- 172.21.15.83:0/2143791006 learned_addr learned my addr 172.21.15.83:0/2143791006 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:30:02.951 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.952+0000 7f64b1115640 1 -- 172.21.15.83:0/2143791006 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1463237205 0 0) 0x7f64ac142520 con 0x7f64ac1430c0 2024-03-25T19:30:02.951 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.952+0000 7f64b1115640 10 monclient(hunting): _negotiate 2024-03-25T19:30:02.952 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.952+0000 7f64b1115640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:02.952 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.952+0000 7f64b1115640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:02.952 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.952+0000 7f64b1115640 10 monclient(hunting): my global_id is 4686 2024-03-25T19:30:02.952 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.952+0000 7f64b1115640 1 -- 172.21.15.83:0/2143791006 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6498003520 con 0x7f64ac1430c0 2024-03-25T19:30:02.958 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.958+0000 7f64b1115640 1 -- 172.21.15.83:0/2143791006 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1334476246 0 0) 0x7f64ac0b28a0 con 0x7f64ac13f710 2024-03-25T19:30:02.958 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.958+0000 7f64b1115640 10 monclient(hunting): _negotiate 2024-03-25T19:30:02.958 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.958+0000 7f64b1115640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:02.958 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.958+0000 7f64b1115640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:02.958 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.958+0000 7f64b1115640 10 monclient(hunting): my global_id is 4702 2024-03-25T19:30:02.958 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.958+0000 7f64b1115640 1 -- 172.21.15.83:0/2143791006 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64ac142520 con 0x7f64ac13f710 2024-03-25T19:30:02.960 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.960+0000 7f64b1115640 1 -- 172.21.15.83:0/2143791006 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3028650650 0 0) 0x7f6498003520 con 0x7f64ac1430c0 2024-03-25T19:30:02.960 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.960+0000 7f64b1115640 1 -- 172.21.15.83:0/2143791006 >> v1:172.21.15.83:6790/0 conn(0x7f64ac0a98f0 legacy=0x7f64ac0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:02.961 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.960+0000 7f64b1115640 1 -- 172.21.15.83:0/2143791006 >> v1:172.21.15.103:6789/0 conn(0x7f64ac13f710 legacy=0x7f64ac1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:02.961 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.960+0000 7f64b1115640 10 monclient: _finish_hunting 0 2024-03-25T19:30:02.961 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.960+0000 7f64b1115640 1 monclient: found mon.noname-a 2024-03-25T19:30:02.961 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.960+0000 7f64b1115640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:02.961 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.960+0000 7f64b1115640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.83:6789/0 2024-03-25T19:30:02.961 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.960+0000 7f64b1115640 1 -- 172.21.15.83:0/2143791006 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64ac0b44c0 con 0x7f64ac1430c0 2024-03-25T19:30:02.961 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.960+0000 7f64b1115640 10 monclient: _finish_auth 0 2024-03-25T19:30:02.961 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.960+0000 7f64b1115640 10 monclient: _check_auth_tickets 2024-03-25T19:30:02.961 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.960+0000 7f64b1115640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:02.961 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.960+0000 7f64b1115640 1 -- 172.21.15.83:0/2143791006 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f64a4001a10 con 0x7f64ac1430c0 2024-03-25T19:30:02.961 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.960+0000 7f64b437b640 5 monclient: authenticate success, global_id 4686 2024-03-25T19:30:02.961 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.960+0000 7f64b437b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:02.961 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.960+0000 7f64b1115640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:02.961 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.960+0000 7f64b1115640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:30:02.961 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.960+0000 7f64b1115640 10 monclient: dump: 2024-03-25T19:30:02.962 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:30:02.962 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:30:02.962 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:02.962 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:02.962 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:02.962 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:30:02.962 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:30:02.962 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:30:02.962 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:30:02.962 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:02.962 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.960+0000 7f64b437b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:02.967 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.967+0000 7f64b1115640 1 -- 172.21.15.83:0/2143791006 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64a4001b70 con 0x7f64ac1430c0 2024-03-25T19:30:02.967 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.967+0000 7f64b1115640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:02.967 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.967+0000 7f64b1115640 1 -- 172.21.15.83:0/2143791006 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f64a4001f80 con 0x7f64ac1430c0 2024-03-25T19:30:02.967 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.967+0000 7f64b437b640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:02.967 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.967+0000 7f64b437b640 4 set_mon_vals no callback set 2024-03-25T19:30:02.967 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.967+0000 7f64b1115640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:02.967 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.967+0000 7f64b1115640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:02.967 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.967+0000 7f64b1115640 10 monclient: dump: 2024-03-25T19:30:02.967 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:30:02.967 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:30:02.967 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:02.967 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:02.967 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:02.968 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:30:02.968 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:30:02.968 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:30:02.968 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:30:02.968 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:02.969 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.968+0000 7f64b437b640 10 monclient: shutdown 2024-03-25T19:30:02.969 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.968+0000 7f64b437b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:02.969 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.968+0000 7f64b437b640 1 -- 172.21.15.83:0/2143791006 >> v1:172.21.15.83:6789/0 conn(0x7f64ac1430c0 legacy=0x7f64ac14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:02.969 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.968+0000 7f64b437b640 1 -- 172.21.15.83:0/2143791006 shutdown_connections 2024-03-25T19:30:02.970 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.968+0000 7f64b437b640 1 -- 172.21.15.83:0/2143791006 >> 172.21.15.83:0/2143791006 conn(0x7f64ac0b7ea0 msgr2=0x7f64ac0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:02.970 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.968+0000 7f64b437b640 1 -- 172.21.15.83:0/2143791006 shutdown_connections 2024-03-25T19:30:02.970 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.969+0000 7f64b437b640 1 -- 172.21.15.83:0/2143791006 wait complete. 2024-03-25T19:30:02.970 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.969+0000 7f64b437b640 10 monclient: build_initial_monmap 2024-03-25T19:30:02.970 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.969+0000 7f64b437b640 10 monclient: monmap: 2024-03-25T19:30:02.970 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:30:02.970 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:02.970 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:30:02.970 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:30:02.970 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:02.970 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:30:02.970 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:30:02.970 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:30:02.971 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:30:02.971 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:02.971 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.970+0000 7f64b437b640 1 librados: starting msgr at 2024-03-25T19:30:02.971 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.970+0000 7f64b437b640 1 librados: starting objecter 2024-03-25T19:30:02.971 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.970+0000 7f64b437b640 1 Processor -- start 2024-03-25T19:30:02.971 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.970+0000 7f64b437b640 1 -- start start 2024-03-25T19:30:02.971 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.970+0000 7f64b437b640 1 librados: setting wanted keys 2024-03-25T19:30:02.971 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.970+0000 7f64b437b640 1 librados: calling monclient init 2024-03-25T19:30:02.971 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.970+0000 7f64b437b640 10 monclient: init 2024-03-25T19:30:02.972 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.970+0000 7f64b437b640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:02.972 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.970+0000 7f64b437b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:30:02.972 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.970+0000 7f64b437b640 10 monclient(hunting): picked mon.noname-b con 0x7f64ac0a98f0 addr v1:172.21.15.103:6789/0 2024-03-25T19:30:02.972 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.970+0000 7f64b437b640 10 monclient(hunting): picked mon.noname-c con 0x7f64ac13f710 addr v1:172.21.15.83:6790/0 2024-03-25T19:30:02.972 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.970+0000 7f64b437b640 10 monclient(hunting): picked mon.noname-a con 0x7f64ac1430c0 addr v1:172.21.15.83:6789/0 2024-03-25T19:30:02.973 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.970+0000 7f64b437b640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64ac1d7df0 con 0x7f64ac1430c0 2024-03-25T19:30:02.973 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.970+0000 7f64b437b640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64ac1d7f80 con 0x7f64ac0a98f0 2024-03-25T19:30:02.973 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.971+0000 7f64b437b640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64ac13b5c0 con 0x7f64ac13f710 2024-03-25T19:30:02.973 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.971+0000 7f64b437b640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:02.973 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.971+0000 7f64b437b640 10 monclient(hunting): authenticate will time out at 6660.242777s 2024-03-25T19:30:02.981 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.981+0000 7f64b2918640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f64ac1430c0 0x7f64ac135090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:58234/0 (socket says 172.21.15.83:58234) 2024-03-25T19:30:02.981 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.981+0000 7f64b2918640 1 -- 172.21.15.83:0/2205998054 learned_addr learned my addr 172.21.15.83:0/2205998054 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:30:02.984 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.985+0000 7f64a2ffd640 10 client.?.objecter ms_handle_connect 0x7f64ac1430c0 2024-03-25T19:30:02.985 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.985+0000 7f64a2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:02.989 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.989+0000 7f64a2ffd640 10 client.?.objecter ms_handle_connect 0x7f64ac13f710 2024-03-25T19:30:02.989 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.989+0000 7f64a2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:02.991 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.991+0000 7f64a2ffd640 1 -- 172.21.15.83:0/2205998054 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3376053586 0 0) 0x7f64ac1d7df0 con 0x7f64ac1430c0 2024-03-25T19:30:02.991 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.991+0000 7f64a2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:30:02.991 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.991+0000 7f64a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:02.991 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.991+0000 7f64a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:02.991 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.991+0000 7f64a2ffd640 10 monclient(hunting): my global_id is 4689 2024-03-25T19:30:02.991 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.991+0000 7f64a2ffd640 1 -- 172.21.15.83:0/2205998054 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6490004eb0 con 0x7f64ac1430c0 2024-03-25T19:30:02.992 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.992+0000 7f64a2ffd640 10 client.?.objecter ms_handle_connect 0x7f64ac0a98f0 2024-03-25T19:30:02.992 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:02.992+0000 7f64a2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:03.001 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.001+0000 7f64a2ffd640 1 -- 172.21.15.83:0/2205998054 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3728968612 0 0) 0x7f64ac1d7f80 con 0x7f64ac0a98f0 2024-03-25T19:30:03.001 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.001+0000 7f64a2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:30:03.001 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.001+0000 7f64a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:03.002 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.001+0000 7f64a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:03.002 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.001+0000 7f64a2ffd640 10 monclient(hunting): my global_id is 4705 2024-03-25T19:30:03.002 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.001+0000 7f64a2ffd640 1 -- 172.21.15.83:0/2205998054 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64ac1d7df0 con 0x7f64ac0a98f0 2024-03-25T19:30:03.005 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.005+0000 7f64a2ffd640 1 -- 172.21.15.83:0/2205998054 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3239006674 0 0) 0x7f64ac13b5c0 con 0x7f64ac13f710 2024-03-25T19:30:03.005 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.005+0000 7f64a2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:30:03.005 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.005+0000 7f64a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:03.005 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.005+0000 7f64a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:03.005 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.005+0000 7f64a2ffd640 10 monclient(hunting): my global_id is 4703 2024-03-25T19:30:03.006 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.005+0000 7f64a2ffd640 1 -- 172.21.15.83:0/2205998054 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64ac1d7f80 con 0x7f64ac13f710 2024-03-25T19:30:03.007 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.007+0000 7f64a2ffd640 1 -- 172.21.15.83:0/2205998054 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3850112832 0 0) 0x7f6490004eb0 con 0x7f64ac1430c0 2024-03-25T19:30:03.007 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.007+0000 7f64a2ffd640 1 -- 172.21.15.83:0/2205998054 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64ac13b5c0 con 0x7f64ac1430c0 2024-03-25T19:30:03.007 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.007+0000 7f64a2ffd640 1 -- 172.21.15.83:0/2205998054 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f64a4001ec0 con 0x7f64ac1430c0 2024-03-25T19:30:03.007 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.007+0000 7f64a2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:03.007 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.007+0000 7f64a2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:30:03.007 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.007+0000 7f64a2ffd640 10 monclient(hunting): dump: 2024-03-25T19:30:03.007 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:30:03.007 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:30:03.007 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:03.007 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:03.008 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:03.008 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:30:03.008 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:30:03.008 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:30:03.008 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:30:03.008 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:03.008 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.007+0000 7f64a2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:03.020 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.019+0000 7f64a2ffd640 1 -- 172.21.15.83:0/2205998054 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 31551888 0 0) 0x7f64ac13b5c0 con 0x7f64ac1430c0 2024-03-25T19:30:03.020 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.019+0000 7f64a2ffd640 1 -- 172.21.15.83:0/2205998054 >> v1:172.21.15.83:6790/0 conn(0x7f64ac13f710 legacy=0x7f64ac133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:03.020 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.019+0000 7f64a2ffd640 1 -- 172.21.15.83:0/2205998054 >> v1:172.21.15.103:6789/0 conn(0x7f64ac0a98f0 legacy=0x7f64ac1d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:03.020 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.019+0000 7f64a2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:30:03.020 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.019+0000 7f64a2ffd640 1 monclient: found mon.a 2024-03-25T19:30:03.020 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.019+0000 7f64a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:03.020 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.019+0000 7f64a2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:30:03.020 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.019+0000 7f64a2ffd640 1 -- 172.21.15.83:0/2205998054 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64ac13b750 con 0x7f64ac1430c0 2024-03-25T19:30:03.020 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.019+0000 7f64a2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:30:03.020 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.019+0000 7f64a2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:30:03.020 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.019+0000 7f64a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:03.020 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.019+0000 7f64b437b640 5 monclient: authenticate success, global_id 4689 2024-03-25T19:30:03.020 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.019+0000 7f64b437b640 10 monclient: _renew_subs 2024-03-25T19:30:03.021 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.019+0000 7f64b437b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:30:03.021 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.019+0000 7f64b437b640 1 -- 172.21.15.83:0/2205998054 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64ac133710 con 0x7f64ac1430c0 2024-03-25T19:30:03.021 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.019+0000 7f64b437b640 10 client.4689.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:03.021 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.020+0000 7f64b437b640 10 monclient: _renew_subs 2024-03-25T19:30:03.021 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.020+0000 7f64b437b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:30:03.021 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.020+0000 7f64b437b640 1 -- 172.21.15.83:0/2205998054 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f64ac1357a0 con 0x7f64ac1430c0 2024-03-25T19:30:03.021 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.020+0000 7f64b437b640 1 librados: init done 2024-03-25T19:30:03.022 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.020+0000 7f64b437b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:03.022 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.020+0000 7f64b437b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:03.022 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.020+0000 7f64b437b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:30:03.022 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.020+0000 7f64b437b640 1 -- 172.21.15.83:0/2205998054 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64ac138fd0 con 0x7f64ac1430c0 2024-03-25T19:30:03.340 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.341+0000 7f64a2ffd640 1 -- 172.21.15.83:0/2205998054 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64a4004e10 con 0x7f64ac1430c0 2024-03-25T19:30:03.341 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.341+0000 7f64a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:03.341 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.341+0000 7f64a2ffd640 1 -- 172.21.15.83:0/2205998054 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f64a4002100 con 0x7f64ac1430c0 2024-03-25T19:30:03.341 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.341+0000 7f64a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:03.341 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.341+0000 7f64a2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:03.341 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.341+0000 7f64a2ffd640 10 monclient: dump: 2024-03-25T19:30:03.341 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:30:03.341 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:30:03.341 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:03.341 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:03.341 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:03.341 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:30:03.341 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:30:03.341 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:30:03.342 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:30:03.342 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:03.342 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.341+0000 7f64b0914640 4 set_mon_vals no callback set 2024-03-25T19:30:03.342 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.342+0000 7f64a2ffd640 1 -- 172.21.15.83:0/2205998054 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7f64a401adf0 con 0x7f64ac1430c0 2024-03-25T19:30:03.342 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.342+0000 7f64b2117640 10 monclient: get_auth_request con 0x7f649006ddf0 auth_method 0 2024-03-25T19:30:03.342 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.343+0000 7f64a2ffd640 10 client.4689.objecter ms_handle_connect 0x7f649006ddf0 2024-03-25T19:30:03.346 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.347+0000 7f64a2ffd640 1 -- 172.21.15.83:0/2205998054 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2659774754 0 0) 0x7f64a40549f0 con 0x7f64ac1430c0 2024-03-25T19:30:03.346 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.347+0000 7f64a2ffd640 10 client.4689.objecter ms_dispatch 0x7f64ac000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:30:03.347 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.347+0000 7f64a2ffd640 3 client.4689.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:30:03.347 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.347+0000 7f64a2ffd640 3 client.4689.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:30:03.347 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.347+0000 7f64a2ffd640 20 client.4689.objecter dump_active .. 0 homeless 2024-03-25T19:30:03.347 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.347+0000 7f64a2ffd640 1 -- 172.21.15.83:0/2205998054 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f64a4050330 con 0x7f64ac1430c0 2024-03-25T19:30:03.347 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.347+0000 7f64a2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:03.347 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.347+0000 7f64a2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:03.563 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.563+0000 7f64b437b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:03.563 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.563+0000 7f64b437b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:03.563 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.563+0000 7f64b437b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:30:03.563 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.563+0000 7f64b437b640 1 -- 172.21.15.83:0/2205998054 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64ac1d8110 con 0x7f64ac1430c0 2024-03-25T19:30:03.655 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.655+0000 7f64a2ffd640 1 -- 172.21.15.83:0/2205998054 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1070525753) 0x7f64a4052430 con 0x7f64ac1430c0 2024-03-25T19:30:03.655 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.655+0000 7f64a2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:03.655 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.655+0000 7f64a2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:03.655 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:30:03.655 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":11,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:29:48.933916+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:48.933916+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:03.658 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.657+0000 7f64b437b640 10 librados: watch_flush enter 2024-03-25T19:30:03.658 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.657+0000 7f64b437b640 10 librados: watch_flush exit 2024-03-25T19:30:03.658 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.657+0000 7f64b437b640 20 client.4689.objecter shutdown clearing up homeless session... 2024-03-25T19:30:03.658 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.657+0000 7f64b437b640 10 client.4689.objecter successfully canceled tick 2024-03-25T19:30:03.658 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.657+0000 7f64b437b640 1 -- 172.21.15.83:0/2205998054 >> v1:172.21.15.103:6800/345072327 conn(0x7f649006ddf0 legacy=0x7f64900702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:03.658 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.657+0000 7f64b437b640 10 monclient: shutdown 2024-03-25T19:30:03.658 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.657+0000 7f64b437b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:03.658 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.657+0000 7f64b437b640 1 -- 172.21.15.83:0/2205998054 >> v1:172.21.15.83:6789/0 conn(0x7f64ac1430c0 legacy=0x7f64ac135090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:03.658 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.658+0000 7f64b437b640 1 -- 172.21.15.83:0/2205998054 shutdown_connections 2024-03-25T19:30:03.658 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.658+0000 7f64b437b640 1 -- 172.21.15.83:0/2205998054 >> 172.21.15.83:0/2205998054 conn(0x7f64ac0b7ea0 msgr2=0x7f64ac074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:03.658 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.658+0000 7f64b437b640 1 -- 172.21.15.83:0/2205998054 shutdown_connections 2024-03-25T19:30:03.658 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.658+0000 7f64b437b640 1 -- 172.21.15.83:0/2205998054 wait complete. 2024-03-25T19:30:03.658 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:03.658+0000 7f64b437b640 1 librados: shutdown 2024-03-25T19:30:03.668 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:09.103 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:10:51.030 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:09.107 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:10:51.034 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:09.141 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:10:51.068 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:09.153 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:10:51.081 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:09.159 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:10:51.086 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:09.167 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:10:51.077 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:09.174 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:10:51.084 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:09.185 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:10:51.112 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:09.196 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:10:51.106 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:10.475 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:10.591 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.591+0000 7f21d0ab5640 10 monclient: get_monmap_and_config 2024-03-25T19:30:10.591 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.591+0000 7f21d0ab5640 10 monclient: build_initial_monmap 2024-03-25T19:30:10.591 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.591+0000 7f21d0ab5640 10 monclient: monmap: 2024-03-25T19:30:10.591 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:30:10.591 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:30:10.591 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:30:10.592340+0000 2024-03-25T19:30:10.591 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:30:10.592340+0000 2024-03-25T19:30:10.591 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:10.591 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:30:10.591 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:30:10.591 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:30:10.591 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:30:10.591 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:10.592 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.592+0000 7f21d0ab5640 1 Processor -- start 2024-03-25T19:30:10.593 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.592+0000 7f21d0ab5640 1 -- start start 2024-03-25T19:30:10.593 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.592+0000 7f21d0ab5640 10 monclient: init 2024-03-25T19:30:10.593 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.593+0000 7f21d0ab5640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:10.593 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.593+0000 7f21d0ab5640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:30:10.593 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.593+0000 7f21d0ab5640 10 monclient(hunting): picked mon.noname-b con 0x7f21cc06ea40 addr v1:172.21.15.83:6790/0 2024-03-25T19:30:10.593 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.593+0000 7f21d0ab5640 10 monclient(hunting): picked mon.noname-c con 0x7f21cc071800 addr v1:172.21.15.103:6789/0 2024-03-25T19:30:10.593 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.593+0000 7f21d0ab5640 10 monclient(hunting): picked mon.noname-a con 0x7f21cc071c40 addr v1:172.21.15.83:6789/0 2024-03-25T19:30:10.593 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.593+0000 7f21d0ab5640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21cc06f1f0 con 0x7f21cc071c40 2024-03-25T19:30:10.593 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.593+0000 7f21d0ab5640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21cc070d00 con 0x7f21cc071800 2024-03-25T19:30:10.593 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.593+0000 7f21d0ab5640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21cc0720b0 con 0x7f21cc06ea40 2024-03-25T19:30:10.593 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.593+0000 7f21d0ab5640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:10.593 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.593+0000 7f21d0ab5640 10 monclient(hunting): authenticate will time out at 6667.864963s 2024-03-25T19:30:10.602 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.603+0000 7f21ca59c640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f21cc06ea40 0x7f21cc13ee90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:36452/0 (socket says 172.21.15.83:36452) 2024-03-25T19:30:10.602 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.603+0000 7f21ca59c640 1 -- 172.21.15.83:0/503379166 learned_addr learned my addr 172.21.15.83:0/503379166 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:30:10.614 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.615+0000 7f21c9d9b640 1 -- 172.21.15.83:0/503379166 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2636167260 0 0) 0x7f21cc0720b0 con 0x7f21cc06ea40 2024-03-25T19:30:10.614 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.615+0000 7f21c9d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:30:10.614 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.615+0000 7f21c9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:10.614 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.615+0000 7f21c9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:10.615 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.615+0000 7f21c9d9b640 10 monclient(hunting): my global_id is 4706 2024-03-25T19:30:10.615 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.615+0000 7f21c9d9b640 1 -- 172.21.15.83:0/503379166 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21b8003520 con 0x7f21cc06ea40 2024-03-25T19:30:10.622 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.622+0000 7f21c9d9b640 1 -- 172.21.15.83:0/503379166 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4151563315 0 0) 0x7f21cc070d00 con 0x7f21cc071800 2024-03-25T19:30:10.622 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.622+0000 7f21c9d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:30:10.622 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.622+0000 7f21c9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:10.622 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.622+0000 7f21c9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:10.622 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.622+0000 7f21c9d9b640 10 monclient(hunting): my global_id is 4708 2024-03-25T19:30:10.622 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.622+0000 7f21c9d9b640 1 -- 172.21.15.83:0/503379166 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21cc0720b0 con 0x7f21cc071800 2024-03-25T19:30:10.625 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.625+0000 7f21c9d9b640 1 -- 172.21.15.83:0/503379166 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 406235387 0 0) 0x7f21cc06f1f0 con 0x7f21cc071c40 2024-03-25T19:30:10.625 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.626+0000 7f21c9d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:30:10.626 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.626+0000 7f21c9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:10.626 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.626+0000 7f21c9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:10.626 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.626+0000 7f21c9d9b640 10 monclient(hunting): my global_id is 4692 2024-03-25T19:30:10.626 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.626+0000 7f21c9d9b640 1 -- 172.21.15.83:0/503379166 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21cc070d00 con 0x7f21cc071c40 2024-03-25T19:30:10.638 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.638+0000 7f21c9d9b640 1 -- 172.21.15.83:0/503379166 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1321088894 0 0) 0x7f21b8003520 con 0x7f21cc06ea40 2024-03-25T19:30:10.638 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.638+0000 7f21c9d9b640 1 -- 172.21.15.83:0/503379166 >> v1:172.21.15.103:6789/0 conn(0x7f21cc071800 legacy=0x7f21cc147070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:10.638 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.638+0000 7f21c9d9b640 1 -- 172.21.15.83:0/503379166 >> v1:172.21.15.83:6789/0 conn(0x7f21cc071c40 legacy=0x7f21cc14a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:10.638 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.638+0000 7f21c9d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:30:10.638 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.638+0000 7f21c9d9b640 1 monclient: found mon.noname-b 2024-03-25T19:30:10.639 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.638+0000 7f21c9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:10.639 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.638+0000 7f21c9d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.83:6790/0 2024-03-25T19:30:10.639 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.638+0000 7f21c9d9b640 1 -- 172.21.15.83:0/503379166 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21cc072780 con 0x7f21cc06ea40 2024-03-25T19:30:10.639 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.638+0000 7f21c9d9b640 10 monclient: _finish_auth 0 2024-03-25T19:30:10.639 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.638+0000 7f21c9d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:30:10.639 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.638+0000 7f21c9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:10.639 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.638+0000 7f21c9d9b640 1 -- 172.21.15.83:0/503379166 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f21bc001690 con 0x7f21cc06ea40 2024-03-25T19:30:10.639 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.638+0000 7f21c9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:10.639 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.638+0000 7f21c9d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:30:10.639 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.638+0000 7f21c9d9b640 10 monclient: dump: 2024-03-25T19:30:10.639 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:30:10.639 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:30:10.639 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:10.639 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:10.639 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:10.639 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:30:10.640 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:30:10.640 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:30:10.640 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:30:10.640 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:10.640 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.638+0000 7f21d0ab5640 5 monclient: authenticate success, global_id 4706 2024-03-25T19:30:10.640 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.638+0000 7f21d0ab5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:10.658 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.659+0000 7f21c9d9b640 1 -- 172.21.15.83:0/503379166 <== mon.2 v1:172.21.15.83:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f21bc0017f0 con 0x7f21cc06ea40 2024-03-25T19:30:10.659 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.659+0000 7f21c9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:10.659 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.659+0000 7f21c9d9b640 1 -- 172.21.15.83:0/503379166 <== mon.2 v1:172.21.15.83:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f21bc001c00 con 0x7f21cc06ea40 2024-03-25T19:30:10.659 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.659+0000 7f21d0ab5640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:10.659 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.659+0000 7f21d0ab5640 4 set_mon_vals no callback set 2024-03-25T19:30:10.659 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.659+0000 7f21c9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:10.659 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.659+0000 7f21c9d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:10.659 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.659+0000 7f21c9d9b640 10 monclient: dump: 2024-03-25T19:30:10.659 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:30:10.659 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:30:10.659 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:10.659 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:10.659 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:10.659 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:30:10.659 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:30:10.659 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:30:10.659 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:30:10.660 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:10.661 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.660+0000 7f21d0ab5640 10 monclient: shutdown 2024-03-25T19:30:10.661 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.660+0000 7f21d0ab5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:10.661 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.660+0000 7f21d0ab5640 1 -- 172.21.15.83:0/503379166 >> v1:172.21.15.83:6790/0 conn(0x7f21cc06ea40 legacy=0x7f21cc13ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:10.661 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.660+0000 7f21d0ab5640 1 -- 172.21.15.83:0/503379166 shutdown_connections 2024-03-25T19:30:10.661 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.660+0000 7f21d0ab5640 1 -- 172.21.15.83:0/503379166 >> 172.21.15.83:0/503379166 conn(0x7f21cc06b630 msgr2=0x7f21cc0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:10.661 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.660+0000 7f21d0ab5640 1 -- 172.21.15.83:0/503379166 shutdown_connections 2024-03-25T19:30:10.661 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.660+0000 7f21d0ab5640 1 -- 172.21.15.83:0/503379166 wait complete. 2024-03-25T19:30:10.661 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.661+0000 7f21d0ab5640 10 monclient: build_initial_monmap 2024-03-25T19:30:10.661 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.661+0000 7f21d0ab5640 10 monclient: monmap: 2024-03-25T19:30:10.661 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:30:10.661 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:10.661 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:30:10.661 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:30:10.662 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:10.662 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:30:10.662 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:30:10.662 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:30:10.662 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:30:10.662 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:10.662 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.661+0000 7f21d0ab5640 1 librados: starting msgr at 2024-03-25T19:30:10.662 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.661+0000 7f21d0ab5640 1 librados: starting objecter 2024-03-25T19:30:10.662 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.661+0000 7f21d0ab5640 1 Processor -- start 2024-03-25T19:30:10.662 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.661+0000 7f21d0ab5640 1 -- start start 2024-03-25T19:30:10.662 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.661+0000 7f21d0ab5640 1 librados: setting wanted keys 2024-03-25T19:30:10.662 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.661+0000 7f21d0ab5640 1 librados: calling monclient init 2024-03-25T19:30:10.662 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.661+0000 7f21d0ab5640 10 monclient: init 2024-03-25T19:30:10.663 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.662+0000 7f21d0ab5640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:10.664 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.662+0000 7f21d0ab5640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:30:10.664 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.662+0000 7f21d0ab5640 10 monclient(hunting): picked mon.noname-a con 0x7f21cc06ea40 addr v1:172.21.15.83:6789/0 2024-03-25T19:30:10.664 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.662+0000 7f21d0ab5640 10 monclient(hunting): picked mon.noname-b con 0x7f21cc071800 addr v1:172.21.15.103:6789/0 2024-03-25T19:30:10.664 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.662+0000 7f21d0ab5640 10 monclient(hunting): picked mon.noname-c con 0x7f21cc071c40 addr v1:172.21.15.83:6790/0 2024-03-25T19:30:10.664 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.662+0000 7f21d0ab5640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21cc137470 con 0x7f21cc06ea40 2024-03-25T19:30:10.664 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.662+0000 7f21d0ab5640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21cc135890 con 0x7f21cc071800 2024-03-25T19:30:10.664 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.662+0000 7f21d0ab5640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21cc1c7f90 con 0x7f21cc071c40 2024-03-25T19:30:10.664 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.662+0000 7f21d0ab5640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:10.664 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.662+0000 7f21d0ab5640 10 monclient(hunting): authenticate will time out at 6667.934546s 2024-03-25T19:30:10.672 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.672+0000 7f21c3fff640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7f21cc071800 0x7f21cc134a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:50270/0 (socket says 172.21.15.83:50270) 2024-03-25T19:30:10.672 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.672+0000 7f21c3fff640 1 -- 172.21.15.83:0/4104466348 learned_addr learned my addr 172.21.15.83:0/4104466348 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:30:10.673 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.673+0000 7f21c2ffd640 10 client.?.objecter ms_handle_connect 0x7f21cc071800 2024-03-25T19:30:10.673 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.673+0000 7f21c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:10.681 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.681+0000 7f21c2ffd640 10 client.?.objecter ms_handle_connect 0x7f21cc06ea40 2024-03-25T19:30:10.681 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.681+0000 7f21c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:10.688 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.688+0000 7f21c2ffd640 10 client.?.objecter ms_handle_connect 0x7f21cc071c40 2024-03-25T19:30:10.688 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.688+0000 7f21c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:10.690 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.690+0000 7f21c2ffd640 1 -- 172.21.15.83:0/4104466348 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2847376669 0 0) 0x7f21cc135890 con 0x7f21cc071800 2024-03-25T19:30:10.690 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.690+0000 7f21c2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:30:10.690 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.690+0000 7f21c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:10.690 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.690+0000 7f21c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:10.690 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.690+0000 7f21c2ffd640 10 monclient(hunting): my global_id is 4711 2024-03-25T19:30:10.690 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.690+0000 7f21c2ffd640 1 -- 172.21.15.83:0/4104466348 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21ac004eb0 con 0x7f21cc071800 2024-03-25T19:30:10.692 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.693+0000 7f21c2ffd640 1 -- 172.21.15.83:0/4104466348 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2770344077 0 0) 0x7f21cc137470 con 0x7f21cc06ea40 2024-03-25T19:30:10.692 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.693+0000 7f21c2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:30:10.692 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.693+0000 7f21c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:10.693 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.693+0000 7f21c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:10.693 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.693+0000 7f21c2ffd640 10 monclient(hunting): my global_id is 4695 2024-03-25T19:30:10.693 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.693+0000 7f21c2ffd640 1 -- 172.21.15.83:0/4104466348 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21cc135890 con 0x7f21cc06ea40 2024-03-25T19:30:10.702 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.702+0000 7f21c2ffd640 1 -- 172.21.15.83:0/4104466348 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1320388552 0 0) 0x7f21cc1c7f90 con 0x7f21cc071c40 2024-03-25T19:30:10.702 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.702+0000 7f21c2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:30:10.702 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.702+0000 7f21c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:10.702 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.702+0000 7f21c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:10.702 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.702+0000 7f21c2ffd640 10 monclient(hunting): my global_id is 4709 2024-03-25T19:30:10.702 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.702+0000 7f21c2ffd640 1 -- 172.21.15.83:0/4104466348 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21cc137470 con 0x7f21cc071c40 2024-03-25T19:30:10.710 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.710+0000 7f21c2ffd640 1 -- 172.21.15.83:0/4104466348 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1614475044 0 0) 0x7f21cc135890 con 0x7f21cc06ea40 2024-03-25T19:30:10.710 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.711+0000 7f21c2ffd640 1 -- 172.21.15.83:0/4104466348 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21cc1c7f90 con 0x7f21cc06ea40 2024-03-25T19:30:10.710 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.711+0000 7f21c2ffd640 1 -- 172.21.15.83:0/4104466348 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f21bc001b00 con 0x7f21cc06ea40 2024-03-25T19:30:10.710 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.711+0000 7f21c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:10.711 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.711+0000 7f21c2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:30:10.711 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.711+0000 7f21c2ffd640 10 monclient(hunting): dump: 2024-03-25T19:30:10.711 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:30:10.711 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:30:10.711 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:10.711 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:10.711 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:10.711 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:30:10.711 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:30:10.711 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:30:10.711 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:30:10.711 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:10.711 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.711+0000 7f21c2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:10.716 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.716+0000 7f21c2ffd640 1 -- 172.21.15.83:0/4104466348 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 715326261 0 0) 0x7f21ac004eb0 con 0x7f21cc071800 2024-03-25T19:30:10.716 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.716+0000 7f21c2ffd640 1 -- 172.21.15.83:0/4104466348 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21cc135890 con 0x7f21cc071800 2024-03-25T19:30:10.716 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.716+0000 7f21c2ffd640 1 -- 172.21.15.83:0/4104466348 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f21b4001a30 con 0x7f21cc071800 2024-03-25T19:30:10.716 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.716+0000 7f21c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:10.716 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.716+0000 7f21c2ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:10.716 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.716+0000 7f21c2ffd640 10 monclient(hunting): dump: 2024-03-25T19:30:10.716 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:30:10.716 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:30:10.716 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:10.717 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:10.717 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:10.717 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:30:10.717 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:30:10.717 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:30:10.717 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:30:10.717 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:10.717 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.716+0000 7f21c2ffd640 1 -- 172.21.15.83:0/4104466348 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4143866160 0 0) 0x7f21cc137470 con 0x7f21cc071c40 2024-03-25T19:30:10.717 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.716+0000 7f21c2ffd640 1 -- 172.21.15.83:0/4104466348 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21ac004eb0 con 0x7f21cc071c40 2024-03-25T19:30:10.717 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.716+0000 7f21c2ffd640 1 -- 172.21.15.83:0/4104466348 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f21c4001bd0 con 0x7f21cc071c40 2024-03-25T19:30:10.717 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.716+0000 7f21c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:10.717 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.716+0000 7f21c2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:10.717 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.716+0000 7f21c2ffd640 10 monclient(hunting): dump: 2024-03-25T19:30:10.717 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:30:10.717 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:30:10.717 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:10.718 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:10.718 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:10.718 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:30:10.718 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:30:10.718 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:30:10.718 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:30:10.718 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:10.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.728+0000 7f21c2ffd640 1 -- 172.21.15.83:0/4104466348 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1359342068 0 0) 0x7f21cc1c7f90 con 0x7f21cc06ea40 2024-03-25T19:30:10.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.728+0000 7f21c2ffd640 1 -- 172.21.15.83:0/4104466348 >> v1:172.21.15.83:6790/0 conn(0x7f21cc071c40 legacy=0x7f21cc135180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:10.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.728+0000 7f21c2ffd640 1 -- 172.21.15.83:0/4104466348 >> v1:172.21.15.103:6789/0 conn(0x7f21cc071800 legacy=0x7f21cc134a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:10.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.728+0000 7f21c2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:30:10.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.728+0000 7f21c2ffd640 1 monclient: found mon.a 2024-03-25T19:30:10.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.728+0000 7f21c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:10.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.728+0000 7f21c2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:30:10.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.728+0000 7f21c2ffd640 1 -- 172.21.15.83:0/4104466348 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21cc0ae470 con 0x7f21cc06ea40 2024-03-25T19:30:10.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.728+0000 7f21c2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:30:10.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.728+0000 7f21c2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:30:10.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.728+0000 7f21c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:10.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.728+0000 7f21d0ab5640 5 monclient: authenticate success, global_id 4695 2024-03-25T19:30:10.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.728+0000 7f21d0ab5640 10 monclient: _renew_subs 2024-03-25T19:30:10.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.728+0000 7f21d0ab5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:30:10.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.728+0000 7f21d0ab5640 1 -- 172.21.15.83:0/4104466348 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21cc1d7fb0 con 0x7f21cc06ea40 2024-03-25T19:30:10.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.728+0000 7f21d0ab5640 10 client.4695.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:10.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.728+0000 7f21d0ab5640 10 monclient: _renew_subs 2024-03-25T19:30:10.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.728+0000 7f21d0ab5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:30:10.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.728+0000 7f21d0ab5640 1 -- 172.21.15.83:0/4104466348 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f21cc1381f0 con 0x7f21cc06ea40 2024-03-25T19:30:10.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.728+0000 7f21d0ab5640 1 librados: init done 2024-03-25T19:30:10.730 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.730+0000 7f21d0ab5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:10.730 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.730+0000 7f21d0ab5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:10.730 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.730+0000 7f21d0ab5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:30:10.730 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:10.730+0000 7f21d0ab5640 1 -- 172.21.15.83:0/4104466348 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21cc1d81a0 con 0x7f21cc06ea40 2024-03-25T19:30:11.077 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:11.077+0000 7f21c2ffd640 1 -- 172.21.15.83:0/4104466348 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f21bc001ca0 con 0x7f21cc06ea40 2024-03-25T19:30:11.077 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:11.077+0000 7f21c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:11.077 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:11.077+0000 7f21c2ffd640 1 -- 172.21.15.83:0/4104466348 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7f21bc004c30 con 0x7f21cc06ea40 2024-03-25T19:30:11.077 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:11.077+0000 7f21c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:11.077 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:11.077+0000 7f21c2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:11.077 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:11.077+0000 7f21c2ffd640 10 monclient: dump: 2024-03-25T19:30:11.078 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:30:11.078 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:30:11.078 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:11.078 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:11.078 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:11.078 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:30:11.078 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:30:11.078 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:30:11.078 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:30:11.078 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:11.078 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:11.077+0000 7f21c959a640 4 set_mon_vals no callback set 2024-03-25T19:30:11.079 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:11.078+0000 7f21c2ffd640 1 -- 172.21.15.83:0/4104466348 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7f21bc01ac00 con 0x7f21cc06ea40 2024-03-25T19:30:11.079 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:11.079+0000 7f21c3fff640 10 monclient: get_auth_request con 0x7f21ac06e0f0 auth_method 0 2024-03-25T19:30:11.079 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:11.079+0000 7f21c2ffd640 10 client.4695.objecter ms_handle_connect 0x7f21ac06e0f0 2024-03-25T19:30:11.084 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:11.084+0000 7f21c2ffd640 1 -- 172.21.15.83:0/4104466348 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2659774754 0 0) 0x7f21bc054800 con 0x7f21cc06ea40 2024-03-25T19:30:11.084 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:11.084+0000 7f21c2ffd640 10 client.4695.objecter ms_dispatch 0x7f21cc000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:30:11.085 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:11.084+0000 7f21c2ffd640 3 client.4695.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:30:11.085 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:11.084+0000 7f21c2ffd640 3 client.4695.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:30:11.085 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:11.084+0000 7f21c2ffd640 20 client.4695.objecter dump_active .. 0 homeless 2024-03-25T19:30:11.085 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:11.084+0000 7f21c2ffd640 1 -- 172.21.15.83:0/4104466348 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f21bc050140 con 0x7f21cc06ea40 2024-03-25T19:30:11.085 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:11.084+0000 7f21c2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:11.085 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:11.084+0000 7f21c2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:11.338 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:11.339+0000 7f21d0ab5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:11.339 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:11.339+0000 7f21d0ab5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:11.339 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:11.339+0000 7f21d0ab5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:30:11.339 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:11.339+0000 7f21d0ab5640 1 -- 172.21.15.83:0/4104466348 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc074d80 con 0x7f21cc06ea40 2024-03-25T19:30:11.380 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:11.380+0000 7f21c2ffd640 1 -- 172.21.15.83:0/4104466348 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1070525753) 0x7f21cc074d80 con 0x7f21cc06ea40 2024-03-25T19:30:11.380 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:11.380+0000 7f21c2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:11.380 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:11.380+0000 7f21c2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:11.380 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:30:11.380 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":11,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:29:48.933916+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:48.933916+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:11.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:11.383+0000 7f21d0ab5640 10 librados: watch_flush enter 2024-03-25T19:30:11.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:11.383+0000 7f21d0ab5640 10 librados: watch_flush exit 2024-03-25T19:30:11.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:11.383+0000 7f21d0ab5640 20 client.4695.objecter shutdown clearing up homeless session... 2024-03-25T19:30:11.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:11.383+0000 7f21d0ab5640 10 client.4695.objecter successfully canceled tick 2024-03-25T19:30:11.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:11.383+0000 7f21d0ab5640 1 -- 172.21.15.83:0/4104466348 >> v1:172.21.15.103:6800/345072327 conn(0x7f21ac06e0f0 legacy=0x7f21ac0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:11.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:11.383+0000 7f21d0ab5640 10 monclient: shutdown 2024-03-25T19:30:11.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:11.383+0000 7f21d0ab5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:11.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:11.383+0000 7f21d0ab5640 1 -- 172.21.15.83:0/4104466348 >> v1:172.21.15.83:6789/0 conn(0x7f21cc06ea40 legacy=0x7f21cc137900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:11.384 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:11.383+0000 7f21d0ab5640 1 -- 172.21.15.83:0/4104466348 shutdown_connections 2024-03-25T19:30:11.384 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:11.383+0000 7f21d0ab5640 1 -- 172.21.15.83:0/4104466348 >> 172.21.15.83:0/4104466348 conn(0x7f21cc06b630 msgr2=0x7f21cc13dd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:11.384 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:11.383+0000 7f21d0ab5640 1 -- 172.21.15.83:0/4104466348 shutdown_connections 2024-03-25T19:30:11.384 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:11.383+0000 7f21d0ab5640 1 -- 172.21.15.83:0/4104466348 wait complete. 2024-03-25T19:30:11.384 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:11.383+0000 7f21d0ab5640 1 librados: shutdown 2024-03-25T19:30:11.394 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:18.201 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:18.316 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.316+0000 7fee3f4fb640 10 monclient: get_monmap_and_config 2024-03-25T19:30:18.316 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.316+0000 7fee3f4fb640 10 monclient: build_initial_monmap 2024-03-25T19:30:18.316 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.316+0000 7fee3f4fb640 10 monclient: monmap: 2024-03-25T19:30:18.316 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:30:18.316 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:30:18.316 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:30:18.317649+0000 2024-03-25T19:30:18.316 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:30:18.317649+0000 2024-03-25T19:30:18.316 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:18.317 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:30:18.317 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:30:18.317 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:30:18.317 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:30:18.317 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:18.318 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.317+0000 7fee3f4fb640 1 Processor -- start 2024-03-25T19:30:18.318 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.317+0000 7fee3f4fb640 1 -- start start 2024-03-25T19:30:18.318 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.317+0000 7fee3f4fb640 10 monclient: init 2024-03-25T19:30:18.318 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.318+0000 7fee3f4fb640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:18.318 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.318+0000 7fee3f4fb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:30:18.318 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.318+0000 7fee3f4fb640 10 monclient(hunting): picked mon.noname-b con 0x7fee380b4d60 addr v1:172.21.15.83:6790/0 2024-03-25T19:30:18.318 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.318+0000 7fee3f4fb640 10 monclient(hunting): picked mon.noname-c con 0x7fee380b1730 addr v1:172.21.15.103:6789/0 2024-03-25T19:30:18.318 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.318+0000 7fee3f4fb640 10 monclient(hunting): picked mon.noname-a con 0x7fee3814be90 addr v1:172.21.15.83:6789/0 2024-03-25T19:30:18.318 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.318+0000 7fee3f4fb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee380af940 con 0x7fee3814be90 2024-03-25T19:30:18.319 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.318+0000 7fee3f4fb640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee380b0a70 con 0x7fee380b1730 2024-03-25T19:30:18.319 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.318+0000 7fee3f4fb640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee380b0d50 con 0x7fee380b4d60 2024-03-25T19:30:18.319 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.318+0000 7fee3f4fb640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:18.319 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.318+0000 7fee3f4fb640 10 monclient(hunting): authenticate will time out at 6675.590200s 2024-03-25T19:30:18.321 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.319+0000 7fee3d297640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fee380b4d60 0x7fee3813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:36956/0 (socket says 172.21.15.83:36956) 2024-03-25T19:30:18.321 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.319+0000 7fee3d297640 1 -- 172.21.15.83:0/3963405132 learned_addr learned my addr 172.21.15.83:0/3963405132 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:30:18.334 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.334+0000 7fee2ffff640 1 -- 172.21.15.83:0/3963405132 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 830733262 0 0) 0x7fee380b0d50 con 0x7fee380b4d60 2024-03-25T19:30:18.334 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.334+0000 7fee2ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:30:18.334 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.334+0000 7fee2ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:18.334 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.334+0000 7fee2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:18.334 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.334+0000 7fee2ffff640 10 monclient(hunting): my global_id is 4712 2024-03-25T19:30:18.334 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.334+0000 7fee2ffff640 1 -- 172.21.15.83:0/3963405132 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee24003520 con 0x7fee380b4d60 2024-03-25T19:30:18.342 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.342+0000 7fee2ffff640 1 -- 172.21.15.83:0/3963405132 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3635707866 0 0) 0x7fee380b0a70 con 0x7fee380b1730 2024-03-25T19:30:18.342 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.342+0000 7fee2ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:30:18.342 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.342+0000 7fee2ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:18.342 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.342+0000 7fee2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:18.342 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.342+0000 7fee2ffff640 10 monclient(hunting): my global_id is 4714 2024-03-25T19:30:18.343 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.342+0000 7fee2ffff640 1 -- 172.21.15.83:0/3963405132 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee380b0d50 con 0x7fee380b1730 2024-03-25T19:30:18.349 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.349+0000 7fee2ffff640 1 -- 172.21.15.83:0/3963405132 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2954508514 0 0) 0x7fee24003520 con 0x7fee380b4d60 2024-03-25T19:30:18.349 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.349+0000 7fee2ffff640 1 -- 172.21.15.83:0/3963405132 >> v1:172.21.15.103:6789/0 conn(0x7fee380b1730 legacy=0x7fee380b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:18.349 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.349+0000 7fee2ffff640 1 -- 172.21.15.83:0/3963405132 >> v1:172.21.15.83:6789/0 conn(0x7fee3814be90 legacy=0x7fee3814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:18.349 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.349+0000 7fee2ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:30:18.349 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.349+0000 7fee2ffff640 1 monclient: found mon.noname-b 2024-03-25T19:30:18.349 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.349+0000 7fee2ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:18.349 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.349+0000 7fee2ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.83:6790/0 2024-03-25T19:30:18.349 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.349+0000 7fee2ffff640 1 -- 172.21.15.83:0/3963405132 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee380b1030 con 0x7fee380b4d60 2024-03-25T19:30:18.349 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.349+0000 7fee2ffff640 10 monclient: _finish_auth 0 2024-03-25T19:30:18.350 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.349+0000 7fee2ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:30:18.350 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.349+0000 7fee2ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:18.350 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.349+0000 7fee2ffff640 1 -- 172.21.15.83:0/3963405132 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fee34001690 con 0x7fee380b4d60 2024-03-25T19:30:18.350 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.349+0000 7fee3f4fb640 5 monclient: authenticate success, global_id 4712 2024-03-25T19:30:18.350 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.349+0000 7fee3f4fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:18.350 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.349+0000 7fee2ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:18.350 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.349+0000 7fee2ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:30:18.350 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.349+0000 7fee2ffff640 10 monclient: dump: 2024-03-25T19:30:18.350 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:30:18.350 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:30:18.350 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:18.350 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:18.350 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:18.350 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:30:18.351 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:30:18.351 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:30:18.351 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:30:18.351 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:18.351 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.349+0000 7fee3f4fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:18.354 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.354+0000 7fee2ffff640 1 -- 172.21.15.83:0/3963405132 <== mon.2 v1:172.21.15.83:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee340017f0 con 0x7fee380b4d60 2024-03-25T19:30:18.354 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.354+0000 7fee2ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:18.354 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.354+0000 7fee3f4fb640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:18.354 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.354+0000 7fee3f4fb640 4 set_mon_vals no callback set 2024-03-25T19:30:18.355 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.355+0000 7fee3f4fb640 10 monclient: shutdown 2024-03-25T19:30:18.355 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.355+0000 7fee3f4fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:18.355 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.355+0000 7fee3f4fb640 1 -- 172.21.15.83:0/3963405132 >> v1:172.21.15.83:6790/0 conn(0x7fee380b4d60 legacy=0x7fee3813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:18.355 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.355+0000 7fee3f4fb640 1 -- 172.21.15.83:0/3963405132 shutdown_connections 2024-03-25T19:30:18.355 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.355+0000 7fee3f4fb640 1 -- 172.21.15.83:0/3963405132 >> 172.21.15.83:0/3963405132 conn(0x7fee380617a0 msgr2=0x7fee38061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:18.355 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.355+0000 7fee3f4fb640 1 -- 172.21.15.83:0/3963405132 shutdown_connections 2024-03-25T19:30:18.355 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.355+0000 7fee3f4fb640 1 -- 172.21.15.83:0/3963405132 wait complete. 2024-03-25T19:30:18.356 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.356+0000 7fee3f4fb640 10 monclient: build_initial_monmap 2024-03-25T19:30:18.356 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.356+0000 7fee3f4fb640 10 monclient: monmap: 2024-03-25T19:30:18.356 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:30:18.356 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:18.357 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:30:18.357 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:30:18.357 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:18.357 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:30:18.357 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:30:18.357 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:30:18.357 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:30:18.357 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:18.357 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.356+0000 7fee3f4fb640 1 librados: starting msgr at 2024-03-25T19:30:18.357 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.356+0000 7fee3f4fb640 1 librados: starting objecter 2024-03-25T19:30:18.357 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.356+0000 7fee3f4fb640 1 Processor -- start 2024-03-25T19:30:18.357 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.357+0000 7fee3f4fb640 1 -- start start 2024-03-25T19:30:18.357 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.357+0000 7fee3f4fb640 1 librados: setting wanted keys 2024-03-25T19:30:18.357 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.357+0000 7fee3f4fb640 1 librados: calling monclient init 2024-03-25T19:30:18.357 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.357+0000 7fee3f4fb640 10 monclient: init 2024-03-25T19:30:18.358 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.357+0000 7fee3f4fb640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:18.358 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.357+0000 7fee3f4fb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:30:18.358 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.357+0000 7fee3f4fb640 10 monclient(hunting): picked mon.noname-b con 0x7fee380b1730 addr v1:172.21.15.103:6789/0 2024-03-25T19:30:18.358 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.357+0000 7fee3f4fb640 10 monclient(hunting): picked mon.noname-a con 0x7fee380b4d60 addr v1:172.21.15.83:6789/0 2024-03-25T19:30:18.358 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.357+0000 7fee3f4fb640 10 monclient(hunting): picked mon.noname-c con 0x7fee3814be90 addr v1:172.21.15.83:6790/0 2024-03-25T19:30:18.359 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.357+0000 7fee3f4fb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee38136090 con 0x7fee380b4d60 2024-03-25T19:30:18.359 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.357+0000 7fee3f4fb640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee381dad30 con 0x7fee380b1730 2024-03-25T19:30:18.359 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.357+0000 7fee3f4fb640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee381db960 con 0x7fee3814be90 2024-03-25T19:30:18.359 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.357+0000 7fee3f4fb640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:18.359 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.357+0000 7fee3f4fb640 10 monclient(hunting): authenticate will time out at 6675.629713s 2024-03-25T19:30:18.364 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.365+0000 7fee3da98640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fee3814be90 0x7fee38136b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:37006/0 (socket says 172.21.15.83:37006) 2024-03-25T19:30:18.364 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.365+0000 7fee3da98640 1 -- 172.21.15.83:0/902433544 learned_addr learned my addr 172.21.15.83:0/902433544 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:30:18.366 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.366+0000 7fee2dffb640 10 client.?.objecter ms_handle_connect 0x7fee380b1730 2024-03-25T19:30:18.366 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.366+0000 7fee2dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:18.369 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.370+0000 7fee2dffb640 10 client.?.objecter ms_handle_connect 0x7fee3814be90 2024-03-25T19:30:18.370 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.370+0000 7fee2dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:18.380 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.380+0000 7fee2dffb640 1 -- 172.21.15.83:0/902433544 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2666773829 0 0) 0x7fee381dad30 con 0x7fee380b1730 2024-03-25T19:30:18.380 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.380+0000 7fee2dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:30:18.380 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.380+0000 7fee2dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:18.380 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.380+0000 7fee2dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:18.380 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.380+0000 7fee2dffb640 10 monclient(hunting): my global_id is 4717 2024-03-25T19:30:18.380 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.380+0000 7fee2dffb640 1 -- 172.21.15.83:0/902433544 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee1c004eb0 con 0x7fee380b1730 2024-03-25T19:30:18.380 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.381+0000 7fee2dffb640 10 client.?.objecter ms_handle_connect 0x7fee380b4d60 2024-03-25T19:30:18.381 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.381+0000 7fee2dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:18.384 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.385+0000 7fee2dffb640 1 -- 172.21.15.83:0/902433544 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4241666266 0 0) 0x7fee381db960 con 0x7fee3814be90 2024-03-25T19:30:18.384 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.385+0000 7fee2dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:30:18.384 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.385+0000 7fee2dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:18.384 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.385+0000 7fee2dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:18.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.385+0000 7fee2dffb640 10 monclient(hunting): my global_id is 4715 2024-03-25T19:30:18.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.385+0000 7fee2dffb640 1 -- 172.21.15.83:0/902433544 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee381dad30 con 0x7fee3814be90 2024-03-25T19:30:18.394 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.395+0000 7fee2dffb640 1 -- 172.21.15.83:0/902433544 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 222333137 0 0) 0x7fee38136090 con 0x7fee380b4d60 2024-03-25T19:30:18.394 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.395+0000 7fee2dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:30:18.395 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.395+0000 7fee2dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:18.395 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.395+0000 7fee2dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:18.395 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.395+0000 7fee2dffb640 10 monclient(hunting): my global_id is 4701 2024-03-25T19:30:18.395 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.395+0000 7fee2dffb640 1 -- 172.21.15.83:0/902433544 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee381db960 con 0x7fee380b4d60 2024-03-25T19:30:18.397 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.397+0000 7fee2dffb640 1 -- 172.21.15.83:0/902433544 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1799865899 0 0) 0x7fee1c004eb0 con 0x7fee380b1730 2024-03-25T19:30:18.397 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.397+0000 7fee2dffb640 1 -- 172.21.15.83:0/902433544 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee38136090 con 0x7fee380b1730 2024-03-25T19:30:18.397 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.397+0000 7fee2dffb640 1 -- 172.21.15.83:0/902433544 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fee34001ca0 con 0x7fee380b1730 2024-03-25T19:30:18.397 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.397+0000 7fee2dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:18.397 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.397+0000 7fee2dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:30:18.397 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.397+0000 7fee2dffb640 10 monclient(hunting): dump: 2024-03-25T19:30:18.397 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:30:18.397 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:30:18.397 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:18.397 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:18.397 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:18.397 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:30:18.398 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:30:18.398 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:30:18.398 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:30:18.398 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:18.398 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.397+0000 7fee2dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:18.398 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.398+0000 7fee2dffb640 1 -- 172.21.15.83:0/902433544 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1142260355 0 0) 0x7fee381dad30 con 0x7fee3814be90 2024-03-25T19:30:18.398 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.398+0000 7fee2dffb640 1 -- 172.21.15.83:0/902433544 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee1c004eb0 con 0x7fee3814be90 2024-03-25T19:30:18.398 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.398+0000 7fee2dffb640 1 -- 172.21.15.83:0/902433544 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fee30001bd0 con 0x7fee3814be90 2024-03-25T19:30:18.399 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.398+0000 7fee2dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:18.399 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.398+0000 7fee2dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:18.399 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.398+0000 7fee2dffb640 10 monclient(hunting): dump: 2024-03-25T19:30:18.399 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:30:18.399 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:30:18.399 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:18.399 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:18.399 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:18.399 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:30:18.399 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:30:18.399 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:30:18.399 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:30:18.399 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:18.409 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.409+0000 7fee2dffb640 1 -- 172.21.15.83:0/902433544 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3165884523 0 0) 0x7fee381db960 con 0x7fee380b4d60 2024-03-25T19:30:18.409 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.409+0000 7fee2dffb640 1 -- 172.21.15.83:0/902433544 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee381dad30 con 0x7fee380b4d60 2024-03-25T19:30:18.409 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.409+0000 7fee2dffb640 1 -- 172.21.15.83:0/902433544 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fee28001930 con 0x7fee380b4d60 2024-03-25T19:30:18.409 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.409+0000 7fee2dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:18.409 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.409+0000 7fee2dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:18.409 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.409+0000 7fee2dffb640 10 monclient(hunting): dump: 2024-03-25T19:30:18.409 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:30:18.409 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:30:18.410 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:18.410 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:18.410 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:18.410 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:30:18.410 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:30:18.410 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:30:18.410 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:30:18.410 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:18.414 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.413+0000 7fee2dffb640 1 -- 172.21.15.83:0/902433544 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3212648368 0 0) 0x7fee1c004eb0 con 0x7fee3814be90 2024-03-25T19:30:18.414 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.413+0000 7fee2dffb640 1 -- 172.21.15.83:0/902433544 >> v1:172.21.15.103:6789/0 conn(0x7fee380b1730 legacy=0x7fee381db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:18.414 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.413+0000 7fee2dffb640 1 -- 172.21.15.83:0/902433544 >> v1:172.21.15.83:6789/0 conn(0x7fee380b4d60 legacy=0x7fee3812fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:18.414 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.413+0000 7fee2dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:30:18.414 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.413+0000 7fee2dffb640 1 monclient: found mon.c 2024-03-25T19:30:18.414 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.413+0000 7fee2dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:18.414 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.413+0000 7fee2dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:30:18.414 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.413+0000 7fee2dffb640 1 -- 172.21.15.83:0/902433544 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee38130570 con 0x7fee3814be90 2024-03-25T19:30:18.414 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.413+0000 7fee2dffb640 10 monclient: _finish_auth 0 2024-03-25T19:30:18.415 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.413+0000 7fee2dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:30:18.415 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.413+0000 7fee2dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:18.415 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.413+0000 7fee3f4fb640 5 monclient: authenticate success, global_id 4715 2024-03-25T19:30:18.415 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.413+0000 7fee3f4fb640 10 monclient: _renew_subs 2024-03-25T19:30:18.415 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.413+0000 7fee3f4fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:30:18.415 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.413+0000 7fee3f4fb640 1 -- 172.21.15.83:0/902433544 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee381307c0 con 0x7fee3814be90 2024-03-25T19:30:18.415 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.413+0000 7fee3f4fb640 10 client.4715.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:18.415 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.413+0000 7fee3f4fb640 10 monclient: _renew_subs 2024-03-25T19:30:18.415 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.413+0000 7fee3f4fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:30:18.415 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.413+0000 7fee3f4fb640 1 -- 172.21.15.83:0/902433544 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fee38143ae0 con 0x7fee3814be90 2024-03-25T19:30:18.415 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.413+0000 7fee3f4fb640 1 librados: init done 2024-03-25T19:30:18.415 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.414+0000 7fee3f4fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:18.416 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.414+0000 7fee3f4fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:18.416 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.414+0000 7fee3f4fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:30:18.416 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.414+0000 7fee3f4fb640 1 -- 172.21.15.83:0/902433544 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee381dab20 con 0x7fee3814be90 2024-03-25T19:30:18.703 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.703+0000 7fee2dffb640 1 -- 172.21.15.83:0/902433544 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee300020a0 con 0x7fee3814be90 2024-03-25T19:30:18.703 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.703+0000 7fee2dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:18.704 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.703+0000 7fee2dffb640 1 -- 172.21.15.83:0/902433544 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fee30003b30 con 0x7fee3814be90 2024-03-25T19:30:18.704 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.703+0000 7fee2dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:18.704 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.703+0000 7fee2dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:18.704 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.703+0000 7fee2dffb640 10 monclient: dump: 2024-03-25T19:30:18.704 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:30:18.704 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:30:18.704 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:18.704 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:18.704 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:18.704 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:30:18.704 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:30:18.704 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:30:18.704 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:30:18.704 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:18.704 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.703+0000 7fee2f7fe640 4 set_mon_vals no callback set 2024-03-25T19:30:18.705 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.704+0000 7fee2dffb640 1 -- 172.21.15.83:0/902433544 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7fee30003d50 con 0x7fee3814be90 2024-03-25T19:30:18.705 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.705+0000 7fee3d297640 10 monclient: get_auth_request con 0x7fee1c06e110 auth_method 0 2024-03-25T19:30:18.705 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.705+0000 7fee2dffb640 10 client.4715.objecter ms_handle_connect 0x7fee1c06e110 2024-03-25T19:30:18.711 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.711+0000 7fee2dffb640 1 -- 172.21.15.83:0/902433544 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2659774754 0 0) 0x7fee300549c0 con 0x7fee3814be90 2024-03-25T19:30:18.711 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.711+0000 7fee2dffb640 10 client.4715.objecter ms_dispatch 0x7fee38000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:30:18.711 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.711+0000 7fee2dffb640 3 client.4715.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:30:18.711 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.711+0000 7fee2dffb640 3 client.4715.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:30:18.711 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.711+0000 7fee2dffb640 20 client.4715.objecter dump_active .. 0 homeless 2024-03-25T19:30:18.711 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.711+0000 7fee2dffb640 1 -- 172.21.15.83:0/902433544 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fee30050300 con 0x7fee3814be90 2024-03-25T19:30:18.711 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.711+0000 7fee2dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:18.711 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.711+0000 7fee2dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:18.956 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.956+0000 7fee3f4fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:18.956 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.956+0000 7fee3f4fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:18.956 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.956+0000 7fee3f4fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:30:18.956 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.956+0000 7fee3f4fb640 1 -- 172.21.15.83:0/902433544 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee380627b0 con 0x7fee3814be90 2024-03-25T19:30:18.982 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.982+0000 7fee2dffb640 1 -- 172.21.15.83:0/902433544 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1070525753) 0x7fee30052400 con 0x7fee3814be90 2024-03-25T19:30:18.982 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.982+0000 7fee2dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:18.982 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.982+0000 7fee2dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:18.982 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:30:18.982 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":11,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:29:48.933916+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:48.933916+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:18.985 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.984+0000 7fee3f4fb640 10 librados: watch_flush enter 2024-03-25T19:30:18.985 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.984+0000 7fee3f4fb640 10 librados: watch_flush exit 2024-03-25T19:30:18.985 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.984+0000 7fee3f4fb640 20 client.4715.objecter shutdown clearing up homeless session... 2024-03-25T19:30:18.985 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.984+0000 7fee3f4fb640 10 client.4715.objecter successfully canceled tick 2024-03-25T19:30:18.985 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.985+0000 7fee3f4fb640 1 -- 172.21.15.83:0/902433544 >> v1:172.21.15.103:6800/345072327 conn(0x7fee1c06e110 legacy=0x7fee1c0705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:18.985 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.985+0000 7fee3f4fb640 10 monclient: shutdown 2024-03-25T19:30:18.986 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.985+0000 7fee3f4fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:18.986 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.985+0000 7fee3f4fb640 1 -- 172.21.15.83:0/902433544 >> v1:172.21.15.83:6790/0 conn(0x7fee3814be90 legacy=0x7fee38136b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:18.986 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.985+0000 7fee3f4fb640 1 -- 172.21.15.83:0/902433544 shutdown_connections 2024-03-25T19:30:18.986 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.985+0000 7fee3f4fb640 1 -- 172.21.15.83:0/902433544 >> 172.21.15.83:0/902433544 conn(0x7fee380617a0 msgr2=0x7fee380b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:18.986 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.985+0000 7fee3f4fb640 1 -- 172.21.15.83:0/902433544 shutdown_connections 2024-03-25T19:30:18.986 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.985+0000 7fee3f4fb640 1 -- 172.21.15.83:0/902433544 wait complete. 2024-03-25T19:30:18.986 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:18.985+0000 7fee3f4fb640 1 librados: shutdown 2024-03-25T19:30:18.997 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:25.803 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:25.921 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.920+0000 7fcfc870e640 10 monclient: get_monmap_and_config 2024-03-25T19:30:25.921 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.920+0000 7fcfc870e640 10 monclient: build_initial_monmap 2024-03-25T19:30:25.921 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.920+0000 7fcfc870e640 10 monclient: monmap: 2024-03-25T19:30:25.921 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:30:25.921 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:30:25.921 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:30:25.922587+0000 2024-03-25T19:30:25.921 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:30:25.922587+0000 2024-03-25T19:30:25.921 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:25.921 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:30:25.922 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:30:25.922 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:30:25.922 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:30:25.922 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:25.923 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.921+0000 7fcfc870e640 1 Processor -- start 2024-03-25T19:30:25.923 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.921+0000 7fcfc870e640 1 -- start start 2024-03-25T19:30:25.923 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.921+0000 7fcfc870e640 10 monclient: init 2024-03-25T19:30:25.923 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.922+0000 7fcfc870e640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:25.923 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.922+0000 7fcfc870e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:30:25.923 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.922+0000 7fcfc870e640 10 monclient(hunting): picked mon.noname-a con 0x7fcfc006ab80 addr v1:172.21.15.83:6789/0 2024-03-25T19:30:25.923 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.922+0000 7fcfc870e640 10 monclient(hunting): picked mon.noname-c con 0x7fcfc006af60 addr v1:172.21.15.103:6789/0 2024-03-25T19:30:25.924 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.922+0000 7fcfc870e640 10 monclient(hunting): picked mon.noname-b con 0x7fcfc00b00b0 addr v1:172.21.15.83:6790/0 2024-03-25T19:30:25.924 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.922+0000 7fcfc870e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfc00b0500 con 0x7fcfc006ab80 2024-03-25T19:30:25.924 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.922+0000 7fcfc870e640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfc013ffa0 con 0x7fcfc006af60 2024-03-25T19:30:25.924 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.922+0000 7fcfc870e640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfc00af520 con 0x7fcfc00b00b0 2024-03-25T19:30:25.924 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.922+0000 7fcfc870e640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:25.924 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.922+0000 7fcfc870e640 10 monclient(hunting): authenticate will time out at 6683.195054s 2024-03-25T19:30:25.932 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.931+0000 7fcfc6cab640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fcfc00b00b0 0x7fcfc014dcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:51594/0 (socket says 172.21.15.83:51594) 2024-03-25T19:30:25.932 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.931+0000 7fcfc6cab640 1 -- 172.21.15.83:0/2888388408 learned_addr learned my addr 172.21.15.83:0/2888388408 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:30:25.954 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.953+0000 7fcfc54a8640 1 -- 172.21.15.83:0/2888388408 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3475575536 0 0) 0x7fcfc00af520 con 0x7fcfc00b00b0 2024-03-25T19:30:25.954 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.953+0000 7fcfc54a8640 10 monclient(hunting): _negotiate 2024-03-25T19:30:25.954 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.953+0000 7fcfc54a8640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:25.954 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.953+0000 7fcfc54a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:25.954 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.953+0000 7fcfc54a8640 10 monclient(hunting): my global_id is 4718 2024-03-25T19:30:25.954 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.953+0000 7fcfc54a8640 1 -- 172.21.15.83:0/2888388408 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfac003520 con 0x7fcfc00b00b0 2024-03-25T19:30:25.960 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.959+0000 7fcfc54a8640 1 -- 172.21.15.83:0/2888388408 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1044949993 0 0) 0x7fcfc00b0500 con 0x7fcfc006ab80 2024-03-25T19:30:25.960 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.959+0000 7fcfc54a8640 10 monclient(hunting): _negotiate 2024-03-25T19:30:25.960 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.959+0000 7fcfc54a8640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:25.960 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.959+0000 7fcfc54a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:25.960 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.959+0000 7fcfc54a8640 10 monclient(hunting): my global_id is 4704 2024-03-25T19:30:25.960 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.959+0000 7fcfc54a8640 1 -- 172.21.15.83:0/2888388408 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfc00af520 con 0x7fcfc006ab80 2024-03-25T19:30:25.965 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.963+0000 7fcfc54a8640 1 -- 172.21.15.83:0/2888388408 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3788677160 0 0) 0x7fcfc013ffa0 con 0x7fcfc006af60 2024-03-25T19:30:25.965 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.963+0000 7fcfc54a8640 10 monclient(hunting): _negotiate 2024-03-25T19:30:25.965 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.963+0000 7fcfc54a8640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:25.965 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.963+0000 7fcfc54a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:25.965 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.963+0000 7fcfc54a8640 10 monclient(hunting): my global_id is 4720 2024-03-25T19:30:25.965 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.963+0000 7fcfc54a8640 1 -- 172.21.15.83:0/2888388408 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfc00b0500 con 0x7fcfc006af60 2024-03-25T19:30:25.965 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.964+0000 7fcfc54a8640 1 -- 172.21.15.83:0/2888388408 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3536474843 0 0) 0x7fcfac003520 con 0x7fcfc00b00b0 2024-03-25T19:30:25.965 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.964+0000 7fcfc54a8640 1 -- 172.21.15.83:0/2888388408 >> v1:172.21.15.103:6789/0 conn(0x7fcfc006af60 legacy=0x7fcfc013f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:25.965 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.964+0000 7fcfc54a8640 1 -- 172.21.15.83:0/2888388408 >> v1:172.21.15.83:6789/0 conn(0x7fcfc006ab80 legacy=0x7fcfc00ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:25.965 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.964+0000 7fcfc54a8640 10 monclient: _finish_hunting 0 2024-03-25T19:30:25.965 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.964+0000 7fcfc54a8640 1 monclient: found mon.noname-b 2024-03-25T19:30:25.965 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.964+0000 7fcfc54a8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:25.965 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.964+0000 7fcfc54a8640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.83:6790/0 2024-03-25T19:30:25.966 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.964+0000 7fcfc54a8640 1 -- 172.21.15.83:0/2888388408 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc0072890 con 0x7fcfc00b00b0 2024-03-25T19:30:25.966 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.964+0000 7fcfc54a8640 10 monclient: _finish_auth 0 2024-03-25T19:30:25.966 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.964+0000 7fcfc54a8640 10 monclient: _check_auth_tickets 2024-03-25T19:30:25.966 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.964+0000 7fcfc54a8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:25.966 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.964+0000 7fcfc870e640 5 monclient: authenticate success, global_id 4718 2024-03-25T19:30:25.966 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.964+0000 7fcfc870e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:25.982 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.981+0000 7fcfc54a8640 1 -- 172.21.15.83:0/2888388408 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fcfb80019d0 con 0x7fcfc00b00b0 2024-03-25T19:30:25.982 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.981+0000 7fcfc54a8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:25.982 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.981+0000 7fcfc54a8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:30:25.982 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.981+0000 7fcfc54a8640 10 monclient: dump: 2024-03-25T19:30:25.982 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:30:25.982 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:30:25.982 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:25.982 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:25.982 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:25.982 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:30:25.983 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:30:25.983 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:30:25.983 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:30:25.983 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:25.983 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.981+0000 7fcfc54a8640 1 -- 172.21.15.83:0/2888388408 <== mon.2 v1:172.21.15.83:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcfb8001c20 con 0x7fcfc00b00b0 2024-03-25T19:30:25.983 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.981+0000 7fcfc870e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:25.983 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.981+0000 7fcfc54a8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:25.983 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.981+0000 7fcfc54a8640 1 -- 172.21.15.83:0/2888388408 <== mon.2 v1:172.21.15.83:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fcfb8001fb0 con 0x7fcfc00b00b0 2024-03-25T19:30:25.983 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.981+0000 7fcfc870e640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:25.983 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.981+0000 7fcfc870e640 4 set_mon_vals no callback set 2024-03-25T19:30:25.983 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.981+0000 7fcfc54a8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:25.983 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.981+0000 7fcfc54a8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:25.983 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.981+0000 7fcfc54a8640 10 monclient: dump: 2024-03-25T19:30:25.983 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:30:25.983 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:30:25.983 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:25.984 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:25.984 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:25.984 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:30:25.984 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:30:25.984 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:30:25.984 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:30:25.984 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:25.986 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.982+0000 7fcfc870e640 10 monclient: shutdown 2024-03-25T19:30:25.986 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.982+0000 7fcfc870e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:25.986 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.982+0000 7fcfc870e640 1 -- 172.21.15.83:0/2888388408 >> v1:172.21.15.83:6790/0 conn(0x7fcfc00b00b0 legacy=0x7fcfc014dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:25.986 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.982+0000 7fcfc870e640 1 -- 172.21.15.83:0/2888388408 shutdown_connections 2024-03-25T19:30:25.986 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.982+0000 7fcfc870e640 1 -- 172.21.15.83:0/2888388408 >> 172.21.15.83:0/2888388408 conn(0x7fcfc00b4a80 msgr2=0x7fcfc00b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:25.986 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.982+0000 7fcfc870e640 1 -- 172.21.15.83:0/2888388408 shutdown_connections 2024-03-25T19:30:25.986 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.982+0000 7fcfc870e640 1 -- 172.21.15.83:0/2888388408 wait complete. 2024-03-25T19:30:25.986 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.983+0000 7fcfc870e640 10 monclient: build_initial_monmap 2024-03-25T19:30:25.986 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.983+0000 7fcfc870e640 10 monclient: monmap: 2024-03-25T19:30:25.986 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:30:25.987 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:25.987 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:30:25.987 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:30:25.987 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:25.987 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:30:25.987 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:30:25.987 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:30:25.987 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:30:25.987 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:25.987 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.984+0000 7fcfc870e640 1 librados: starting msgr at 2024-03-25T19:30:25.987 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.984+0000 7fcfc870e640 1 librados: starting objecter 2024-03-25T19:30:25.987 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.984+0000 7fcfc870e640 1 Processor -- start 2024-03-25T19:30:25.987 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.984+0000 7fcfc870e640 1 -- start start 2024-03-25T19:30:25.987 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.984+0000 7fcfc870e640 1 librados: setting wanted keys 2024-03-25T19:30:25.987 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.984+0000 7fcfc870e640 1 librados: calling monclient init 2024-03-25T19:30:25.987 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.984+0000 7fcfc870e640 10 monclient: init 2024-03-25T19:30:25.988 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.984+0000 7fcfc870e640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:25.988 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.984+0000 7fcfc870e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:30:25.988 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.984+0000 7fcfc870e640 10 monclient(hunting): picked mon.noname-a con 0x7fcfc006ab80 addr v1:172.21.15.83:6789/0 2024-03-25T19:30:25.988 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.984+0000 7fcfc870e640 10 monclient(hunting): picked mon.noname-c con 0x7fcfc00b00b0 addr v1:172.21.15.83:6790/0 2024-03-25T19:30:25.988 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.984+0000 7fcfc870e640 10 monclient(hunting): picked mon.noname-b con 0x7fcfc0140c90 addr v1:172.21.15.103:6789/0 2024-03-25T19:30:25.988 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.984+0000 7fcfc870e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfc01d7b60 con 0x7fcfc006ab80 2024-03-25T19:30:25.988 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.984+0000 7fcfc870e640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfc01d4f20 con 0x7fcfc0140c90 2024-03-25T19:30:25.988 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.984+0000 7fcfc870e640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfc01d50b0 con 0x7fcfc00b00b0 2024-03-25T19:30:25.988 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.984+0000 7fcfc870e640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:25.988 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.984+0000 7fcfc870e640 10 monclient(hunting): authenticate will time out at 6683.257720s 2024-03-25T19:30:25.990 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.990+0000 7fcfc6cab640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fcfc0140c90 0x7fcfc0142730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:46974/0 (socket says 172.21.15.83:46974) 2024-03-25T19:30:25.991 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.990+0000 7fcfc6cab640 1 -- 172.21.15.83:0/1302226598 learned_addr learned my addr 172.21.15.83:0/1302226598 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:30:25.991 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.990+0000 7fcfb6ffd640 10 client.?.objecter ms_handle_connect 0x7fcfc0140c90 2024-03-25T19:30:25.991 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.990+0000 7fcfb6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:26.000 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.999+0000 7fcfb6ffd640 1 -- 172.21.15.83:0/1302226598 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3304788411 0 0) 0x7fcfc01d4f20 con 0x7fcfc0140c90 2024-03-25T19:30:26.000 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.999+0000 7fcfb6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:30:26.000 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.999+0000 7fcfb6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:26.000 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.999+0000 7fcfb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:26.000 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.999+0000 7fcfb6ffd640 10 monclient(hunting): my global_id is 4723 2024-03-25T19:30:26.000 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:25.999+0000 7fcfb6ffd640 1 -- 172.21.15.83:0/1302226598 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfa4004eb0 con 0x7fcfc0140c90 2024-03-25T19:30:26.001 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.001+0000 7fcfb6ffd640 10 client.?.objecter ms_handle_connect 0x7fcfc00b00b0 2024-03-25T19:30:26.001 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.001+0000 7fcfb6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:26.008 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.007+0000 7fcfb6ffd640 10 client.?.objecter ms_handle_connect 0x7fcfc006ab80 2024-03-25T19:30:26.008 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.007+0000 7fcfb6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:26.015 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.015+0000 7fcfb6ffd640 1 -- 172.21.15.83:0/1302226598 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3793124914 0 0) 0x7fcfc00b0500 con 0x7fcfc00b00b0 2024-03-25T19:30:26.016 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.015+0000 7fcfb6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:30:26.016 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.015+0000 7fcfb6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:26.016 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.015+0000 7fcfb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:26.016 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.015+0000 7fcfb6ffd640 10 monclient(hunting): my global_id is 4721 2024-03-25T19:30:26.016 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.015+0000 7fcfb6ffd640 1 -- 172.21.15.83:0/1302226598 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfc01d4f20 con 0x7fcfc00b00b0 2024-03-25T19:30:26.018 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.018+0000 7fcfb6ffd640 1 -- 172.21.15.83:0/1302226598 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1295305156 0 0) 0x7fcfa4004eb0 con 0x7fcfc0140c90 2024-03-25T19:30:26.018 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.018+0000 7fcfb6ffd640 1 -- 172.21.15.83:0/1302226598 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcfc00b0500 con 0x7fcfc0140c90 2024-03-25T19:30:26.018 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.018+0000 7fcfb6ffd640 1 -- 172.21.15.83:0/1302226598 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fcfb8001f30 con 0x7fcfc0140c90 2024-03-25T19:30:26.018 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.018+0000 7fcfb6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:26.019 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.018+0000 7fcfb6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:30:26.019 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.018+0000 7fcfb6ffd640 10 monclient(hunting): dump: 2024-03-25T19:30:26.019 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:30:26.019 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:30:26.019 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:26.019 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:26.019 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:26.019 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:30:26.019 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:30:26.019 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:30:26.019 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:30:26.019 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:26.019 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.018+0000 7fcfb6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:26.021 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.020+0000 7fcfb6ffd640 1 -- 172.21.15.83:0/1302226598 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1729969794 0 0) 0x7fcfc01d7b60 con 0x7fcfc006ab80 2024-03-25T19:30:26.021 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.020+0000 7fcfb6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:30:26.021 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.020+0000 7fcfb6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:26.021 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.020+0000 7fcfb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:26.021 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.020+0000 7fcfb6ffd640 10 monclient(hunting): my global_id is 4707 2024-03-25T19:30:26.021 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.020+0000 7fcfb6ffd640 1 -- 172.21.15.83:0/1302226598 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfa4004eb0 con 0x7fcfc006ab80 2024-03-25T19:30:26.024 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.023+0000 7fcfb6ffd640 1 -- 172.21.15.83:0/1302226598 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2976281543 0 0) 0x7fcfc01d4f20 con 0x7fcfc00b00b0 2024-03-25T19:30:26.024 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.023+0000 7fcfb6ffd640 1 -- 172.21.15.83:0/1302226598 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcfc01d7b60 con 0x7fcfc00b00b0 2024-03-25T19:30:26.024 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.023+0000 7fcfb6ffd640 1 -- 172.21.15.83:0/1302226598 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fcfb00016d0 con 0x7fcfc00b00b0 2024-03-25T19:30:26.024 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.023+0000 7fcfb6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:26.024 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.023+0000 7fcfb6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:26.024 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.023+0000 7fcfb6ffd640 10 monclient(hunting): dump: 2024-03-25T19:30:26.024 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:30:26.024 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:30:26.024 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:26.024 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:26.024 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:26.024 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:30:26.025 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:30:26.025 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:30:26.025 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:30:26.025 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:26.037 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.036+0000 7fcfb6ffd640 1 -- 172.21.15.83:0/1302226598 <== mon.1 v1:172.21.15.103:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 643369267 0 0) 0x7fcfc00b0500 con 0x7fcfc0140c90 2024-03-25T19:30:26.037 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.036+0000 7fcfb6ffd640 1 -- 172.21.15.83:0/1302226598 >> v1:172.21.15.83:6790/0 conn(0x7fcfc00b00b0 legacy=0x7fcfc01d4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:26.037 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.036+0000 7fcfb6ffd640 1 -- 172.21.15.83:0/1302226598 >> v1:172.21.15.83:6789/0 conn(0x7fcfc006ab80 legacy=0x7fcfc01d7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:26.037 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.036+0000 7fcfb6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:30:26.037 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.036+0000 7fcfb6ffd640 1 monclient: found mon.b 2024-03-25T19:30:26.037 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.036+0000 7fcfb6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:26.037 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.036+0000 7fcfb6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:30:26.038 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.036+0000 7fcfb6ffd640 1 -- 172.21.15.83:0/1302226598 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc0141170 con 0x7fcfc0140c90 2024-03-25T19:30:26.038 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.036+0000 7fcfb6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:30:26.038 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.036+0000 7fcfb6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:30:26.038 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.036+0000 7fcfb6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:26.038 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.036+0000 7fcfc870e640 5 monclient: authenticate success, global_id 4723 2024-03-25T19:30:26.038 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.036+0000 7fcfc870e640 10 monclient: _renew_subs 2024-03-25T19:30:26.038 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.036+0000 7fcfc870e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:30:26.038 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.036+0000 7fcfc870e640 1 -- 172.21.15.83:0/1302226598 --> v1:172.21.15.103:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfc01401c0 con 0x7fcfc0140c90 2024-03-25T19:30:26.038 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.036+0000 7fcfc870e640 10 client.4723.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:26.038 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.036+0000 7fcfc870e640 10 monclient: _renew_subs 2024-03-25T19:30:26.038 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.036+0000 7fcfc870e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:30:26.038 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.036+0000 7fcfc870e640 1 -- 172.21.15.83:0/1302226598 --> v1:172.21.15.103:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfc0143020 con 0x7fcfc0140c90 2024-03-25T19:30:26.038 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.036+0000 7fcfc870e640 1 librados: init done 2024-03-25T19:30:26.039 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.037+0000 7fcfc870e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:26.039 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.037+0000 7fcfc870e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:26.039 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.037+0000 7fcfc870e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:30:26.039 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.037+0000 7fcfc870e640 1 -- 172.21.15.83:0/1302226598 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfc0142430 con 0x7fcfc0140c90 2024-03-25T19:30:26.314 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.312+0000 7fcfb6ffd640 1 -- 172.21.15.83:0/1302226598 <== mon.1 v1:172.21.15.103:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcfb80015b0 con 0x7fcfc0140c90 2024-03-25T19:30:26.314 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.312+0000 7fcfb6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:26.314 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.312+0000 7fcfb6ffd640 1 -- 172.21.15.83:0/1302226598 <== mon.1 v1:172.21.15.103:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fcfb8004fb0 con 0x7fcfc0140c90 2024-03-25T19:30:26.314 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.312+0000 7fcfb6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:26.314 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.312+0000 7fcfb6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:26.314 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.312+0000 7fcfb6ffd640 10 monclient: dump: 2024-03-25T19:30:26.314 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:30:26.314 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:30:26.314 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:26.314 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:26.314 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:26.314 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:30:26.315 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:30:26.315 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:30:26.315 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:30:26.315 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:26.315 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.312+0000 7fcfc4ca7640 4 set_mon_vals no callback set 2024-03-25T19:30:26.315 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.314+0000 7fcfb6ffd640 1 -- 172.21.15.83:0/1302226598 <== mon.1 v1:172.21.15.103:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7fcfb801af80 con 0x7fcfc0140c90 2024-03-25T19:30:26.315 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.314+0000 7fcfc64aa640 10 monclient: get_auth_request con 0x7fcfa406df20 auth_method 0 2024-03-25T19:30:26.315 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.314+0000 7fcfb6ffd640 10 client.4723.objecter ms_handle_connect 0x7fcfa406df20 2024-03-25T19:30:26.321 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.320+0000 7fcfb6ffd640 1 -- 172.21.15.83:0/1302226598 <== mon.1 v1:172.21.15.103:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2659774754 0 0) 0x7fcfb8054c80 con 0x7fcfc0140c90 2024-03-25T19:30:26.321 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.320+0000 7fcfb6ffd640 10 client.4723.objecter ms_dispatch 0x7fcfc0000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:30:26.321 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.320+0000 7fcfb6ffd640 3 client.4723.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:30:26.321 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.320+0000 7fcfb6ffd640 3 client.4723.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:30:26.321 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.320+0000 7fcfb6ffd640 20 client.4723.objecter dump_active .. 0 homeless 2024-03-25T19:30:26.322 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.321+0000 7fcfb6ffd640 1 -- 172.21.15.83:0/1302226598 <== mon.1 v1:172.21.15.103:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcfb80505c0 con 0x7fcfc0140c90 2024-03-25T19:30:26.322 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.321+0000 7fcfb6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:26.322 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.321+0000 7fcfb6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:26.578 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.577+0000 7fcfc870e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:26.578 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.578+0000 7fcfc870e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:26.579 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.578+0000 7fcfc870e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:30:26.579 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.578+0000 7fcfc870e640 1 -- 172.21.15.83:0/1302226598 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcfc0141300 con 0x7fcfc0140c90 2024-03-25T19:30:26.619 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.618+0000 7fcfb6ffd640 1 -- 172.21.15.83:0/1302226598 <== mon.1 v1:172.21.15.103:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1070525753) 0x7fcfb80526c0 con 0x7fcfc0140c90 2024-03-25T19:30:26.619 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.618+0000 7fcfb6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:26.619 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.618+0000 7fcfb6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:26.619 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:30:26.619 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":11,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:29:48.933916+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:48.933916+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:26.621 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.620+0000 7fcfc870e640 10 librados: watch_flush enter 2024-03-25T19:30:26.622 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.620+0000 7fcfc870e640 10 librados: watch_flush exit 2024-03-25T19:30:26.622 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.620+0000 7fcfc870e640 20 client.4723.objecter shutdown clearing up homeless session... 2024-03-25T19:30:26.622 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.620+0000 7fcfc870e640 10 client.4723.objecter successfully canceled tick 2024-03-25T19:30:26.622 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.620+0000 7fcfc870e640 1 -- 172.21.15.83:0/1302226598 >> v1:172.21.15.103:6800/345072327 conn(0x7fcfa406df20 legacy=0x7fcfa40703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:26.622 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.620+0000 7fcfc870e640 10 monclient: shutdown 2024-03-25T19:30:26.622 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.620+0000 7fcfc870e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:26.622 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.620+0000 7fcfc870e640 1 -- 172.21.15.83:0/1302226598 >> v1:172.21.15.103:6789/0 conn(0x7fcfc0140c90 legacy=0x7fcfc0142730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:26.622 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.620+0000 7fcfc870e640 1 -- 172.21.15.83:0/1302226598 shutdown_connections 2024-03-25T19:30:26.622 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.620+0000 7fcfc870e640 1 -- 172.21.15.83:0/1302226598 >> 172.21.15.83:0/1302226598 conn(0x7fcfc00b4a80 msgr2=0x7fcfc00b4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:26.622 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.620+0000 7fcfc870e640 1 -- 172.21.15.83:0/1302226598 shutdown_connections 2024-03-25T19:30:26.622 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.620+0000 7fcfc870e640 1 -- 172.21.15.83:0/1302226598 wait complete. 2024-03-25T19:30:26.622 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:26.621+0000 7fcfc870e640 1 librados: shutdown 2024-03-25T19:30:26.633 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:29.229 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:11:11.156 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:29.233 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:11:11.160 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:29.269 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:11:11.196 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:29.284 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:11:11.194 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:29.284 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:11:11.212 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:29.289 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:11:11.199 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:29.293 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:11:11.221 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:29.316 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:11:11.226 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:29.331 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:11:11.258 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:33.439 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:33.539 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.540+0000 7fc7d02cf640 10 monclient: get_monmap_and_config 2024-03-25T19:30:33.539 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.540+0000 7fc7d02cf640 10 monclient: build_initial_monmap 2024-03-25T19:30:33.539 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.540+0000 7fc7d02cf640 10 monclient: monmap: 2024-03-25T19:30:33.539 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:30:33.540 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:30:33.540 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:30:33.540939+0000 2024-03-25T19:30:33.540 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:30:33.540939+0000 2024-03-25T19:30:33.540 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:33.540 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:30:33.540 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:30:33.540 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:30:33.540 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:30:33.540 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:33.540 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.540+0000 7fc7d02cf640 1 Processor -- start 2024-03-25T19:30:33.541 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.540+0000 7fc7d02cf640 1 -- start start 2024-03-25T19:30:33.541 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.540+0000 7fc7d02cf640 10 monclient: init 2024-03-25T19:30:33.542 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.541+0000 7fc7d02cf640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:33.542 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.541+0000 7fc7d02cf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:30:33.542 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.541+0000 7fc7d02cf640 10 monclient(hunting): picked mon.noname-a con 0x7fc7c80b4d60 addr v1:172.21.15.83:6789/0 2024-03-25T19:30:33.542 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.541+0000 7fc7d02cf640 10 monclient(hunting): picked mon.noname-c con 0x7fc7c80b1730 addr v1:172.21.15.103:6789/0 2024-03-25T19:30:33.542 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.541+0000 7fc7d02cf640 10 monclient(hunting): picked mon.noname-b con 0x7fc7c814be90 addr v1:172.21.15.83:6790/0 2024-03-25T19:30:33.542 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.541+0000 7fc7d02cf640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7c80af940 con 0x7fc7c80b4d60 2024-03-25T19:30:33.542 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.542+0000 7fc7d02cf640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7c80b0a70 con 0x7fc7c80b1730 2024-03-25T19:30:33.542 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.542+0000 7fc7d02cf640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7c80b0d50 con 0x7fc7c814be90 2024-03-25T19:30:33.542 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.542+0000 7fc7d02cf640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:33.542 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.542+0000 7fc7d02cf640 10 monclient(hunting): authenticate will time out at 6690.813778s 2024-03-25T19:30:33.554 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.554+0000 7fc7cd86a640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fc7c80b1730 0x7fc7c80b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:51420/0 (socket says 172.21.15.83:51420) 2024-03-25T19:30:33.554 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.554+0000 7fc7cd86a640 1 -- 172.21.15.83:0/76709985 learned_addr learned my addr 172.21.15.83:0/76709985 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:30:33.576 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.576+0000 7fc7cd069640 1 -- 172.21.15.83:0/76709985 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 203476574 0 0) 0x7fc7c80b0a70 con 0x7fc7c80b1730 2024-03-25T19:30:33.576 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.576+0000 7fc7cd069640 10 monclient(hunting): _negotiate 2024-03-25T19:30:33.576 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.576+0000 7fc7cd069640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:33.576 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.576+0000 7fc7cd069640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:33.576 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.576+0000 7fc7cd069640 10 monclient(hunting): my global_id is 4726 2024-03-25T19:30:33.576 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.576+0000 7fc7cd069640 1 -- 172.21.15.83:0/76709985 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7b4003520 con 0x7fc7c80b1730 2024-03-25T19:30:33.586 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.586+0000 7fc7cd069640 1 -- 172.21.15.83:0/76709985 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3898089593 0 0) 0x7fc7b4003520 con 0x7fc7c80b1730 2024-03-25T19:30:33.587 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.586+0000 7fc7cd069640 1 -- 172.21.15.83:0/76709985 >> v1:172.21.15.83:6790/0 conn(0x7fc7c814be90 legacy=0x7fc7c814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:30:33.587 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.586+0000 7fc7cd069640 1 -- 172.21.15.83:0/76709985 >> v1:172.21.15.83:6789/0 conn(0x7fc7c80b4d60 legacy=0x7fc7c813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:33.587 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.586+0000 7fc7cd069640 10 monclient: _finish_hunting 0 2024-03-25T19:30:33.587 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.586+0000 7fc7cd069640 1 monclient: found mon.noname-c 2024-03-25T19:30:33.587 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.586+0000 7fc7cd069640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:33.587 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.586+0000 7fc7cd069640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.103:6789/0 2024-03-25T19:30:33.587 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.586+0000 7fc7cd069640 1 -- 172.21.15.83:0/76709985 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7c80b1030 con 0x7fc7c80b1730 2024-03-25T19:30:33.587 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.586+0000 7fc7cd069640 10 monclient: _finish_auth 0 2024-03-25T19:30:33.587 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.586+0000 7fc7cd069640 10 monclient: _check_auth_tickets 2024-03-25T19:30:33.587 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.586+0000 7fc7cd069640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:33.587 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.586+0000 7fc7cd069640 1 -- 172.21.15.83:0/76709985 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fc7bc001690 con 0x7fc7c80b1730 2024-03-25T19:30:33.587 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.586+0000 7fc7cd069640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:33.587 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.586+0000 7fc7cd069640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:30:33.587 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.586+0000 7fc7cd069640 10 monclient: dump: 2024-03-25T19:30:33.588 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:30:33.588 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:30:33.588 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:33.588 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:33.588 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:33.588 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:30:33.588 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:30:33.588 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:30:33.588 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:30:33.588 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:33.588 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.586+0000 7fc7d02cf640 5 monclient: authenticate success, global_id 4726 2024-03-25T19:30:33.588 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.586+0000 7fc7d02cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:33.601 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.602+0000 7fc7cd069640 1 -- 172.21.15.83:0/76709985 <== mon.1 v1:172.21.15.103:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc7bc001860 con 0x7fc7c80b1730 2024-03-25T19:30:33.601 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.602+0000 7fc7cd069640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:33.602 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.602+0000 7fc7cd069640 1 -- 172.21.15.83:0/76709985 <== mon.1 v1:172.21.15.103:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fc7bc001c70 con 0x7fc7c80b1730 2024-03-25T19:30:33.602 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.602+0000 7fc7d02cf640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:33.602 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.602+0000 7fc7d02cf640 4 set_mon_vals no callback set 2024-03-25T19:30:33.602 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.602+0000 7fc7cd069640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:33.602 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.602+0000 7fc7cd069640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:33.602 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.602+0000 7fc7cd069640 10 monclient: dump: 2024-03-25T19:30:33.602 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:30:33.602 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:30:33.602 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:33.602 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:33.602 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:33.602 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:30:33.602 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:30:33.602 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:30:33.602 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:30:33.602 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:33.604 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.602+0000 7fc7d02cf640 10 monclient: shutdown 2024-03-25T19:30:33.605 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.602+0000 7fc7d02cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:33.605 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.602+0000 7fc7d02cf640 1 -- 172.21.15.83:0/76709985 >> v1:172.21.15.103:6789/0 conn(0x7fc7c80b1730 legacy=0x7fc7c80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:33.605 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.603+0000 7fc7d02cf640 1 -- 172.21.15.83:0/76709985 shutdown_connections 2024-03-25T19:30:33.605 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.603+0000 7fc7d02cf640 1 -- 172.21.15.83:0/76709985 >> 172.21.15.83:0/76709985 conn(0x7fc7c80617a0 msgr2=0x7fc7c8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:33.605 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.603+0000 7fc7d02cf640 1 -- 172.21.15.83:0/76709985 shutdown_connections 2024-03-25T19:30:33.605 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.603+0000 7fc7d02cf640 1 -- 172.21.15.83:0/76709985 wait complete. 2024-03-25T19:30:33.605 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.603+0000 7fc7d02cf640 10 monclient: build_initial_monmap 2024-03-25T19:30:33.605 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.603+0000 7fc7d02cf640 10 monclient: monmap: 2024-03-25T19:30:33.605 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:30:33.605 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:33.605 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:30:33.605 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:30:33.605 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:33.606 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:30:33.606 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:30:33.606 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:30:33.606 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:30:33.606 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:33.606 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.604+0000 7fc7d02cf640 1 librados: starting msgr at 2024-03-25T19:30:33.606 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.604+0000 7fc7d02cf640 1 librados: starting objecter 2024-03-25T19:30:33.606 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.604+0000 7fc7d02cf640 1 Processor -- start 2024-03-25T19:30:33.606 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.604+0000 7fc7d02cf640 1 -- start start 2024-03-25T19:30:33.606 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.604+0000 7fc7d02cf640 1 librados: setting wanted keys 2024-03-25T19:30:33.606 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.604+0000 7fc7d02cf640 1 librados: calling monclient init 2024-03-25T19:30:33.606 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.604+0000 7fc7d02cf640 10 monclient: init 2024-03-25T19:30:33.606 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.604+0000 7fc7d02cf640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:33.606 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.604+0000 7fc7d02cf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:30:33.606 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.604+0000 7fc7d02cf640 10 monclient(hunting): picked mon.noname-a con 0x7fc7c80b1730 addr v1:172.21.15.83:6789/0 2024-03-25T19:30:33.606 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.605+0000 7fc7d02cf640 10 monclient(hunting): picked mon.noname-c con 0x7fc7c80b4d60 addr v1:172.21.15.83:6790/0 2024-03-25T19:30:33.607 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.605+0000 7fc7d02cf640 10 monclient(hunting): picked mon.noname-b con 0x7fc7c814be90 addr v1:172.21.15.103:6789/0 2024-03-25T19:30:33.607 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.605+0000 7fc7d02cf640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7c8140980 con 0x7fc7c80b1730 2024-03-25T19:30:33.607 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.605+0000 7fc7d02cf640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7c8137b10 con 0x7fc7c814be90 2024-03-25T19:30:33.607 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.605+0000 7fc7d02cf640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7c8138360 con 0x7fc7c80b4d60 2024-03-25T19:30:33.607 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.605+0000 7fc7d02cf640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:33.607 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.605+0000 7fc7d02cf640 10 monclient(hunting): authenticate will time out at 6690.876918s 2024-03-25T19:30:33.614 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.614+0000 7fc7ce06b640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fc7c80b1730 0x7fc7c8131e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:48044/0 (socket says 172.21.15.83:48044) 2024-03-25T19:30:33.614 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.614+0000 7fc7ce06b640 1 -- 172.21.15.83:0/553216920 learned_addr learned my addr 172.21.15.83:0/553216920 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:30:33.619 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.620+0000 7fc7baffd640 10 client.?.objecter ms_handle_connect 0x7fc7c80b1730 2024-03-25T19:30:33.620 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.620+0000 7fc7baffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:33.621 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.622+0000 7fc7baffd640 10 client.?.objecter ms_handle_connect 0x7fc7c80b4d60 2024-03-25T19:30:33.621 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.622+0000 7fc7baffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:33.630 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.631+0000 7fc7baffd640 1 -- 172.21.15.83:0/553216920 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2681727937 0 0) 0x7fc7c8140980 con 0x7fc7c80b1730 2024-03-25T19:30:33.630 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.631+0000 7fc7baffd640 10 monclient(hunting): _negotiate 2024-03-25T19:30:33.630 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.631+0000 7fc7baffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:33.631 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.631+0000 7fc7baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:33.631 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.631+0000 7fc7baffd640 10 monclient(hunting): my global_id is 4713 2024-03-25T19:30:33.631 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.631+0000 7fc7baffd640 1 -- 172.21.15.83:0/553216920 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7ac004eb0 con 0x7fc7c80b1730 2024-03-25T19:30:33.636 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.636+0000 7fc7baffd640 10 client.?.objecter ms_handle_connect 0x7fc7c814be90 2024-03-25T19:30:33.636 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.636+0000 7fc7baffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:33.645 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.645+0000 7fc7baffd640 1 -- 172.21.15.83:0/553216920 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3255939287 0 0) 0x7fc7c8138360 con 0x7fc7c80b4d60 2024-03-25T19:30:33.645 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.645+0000 7fc7baffd640 10 monclient(hunting): _negotiate 2024-03-25T19:30:33.645 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.645+0000 7fc7baffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:33.645 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.645+0000 7fc7baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:33.645 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.645+0000 7fc7baffd640 10 monclient(hunting): my global_id is 4724 2024-03-25T19:30:33.645 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.645+0000 7fc7baffd640 1 -- 172.21.15.83:0/553216920 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7c8140980 con 0x7fc7c80b4d60 2024-03-25T19:30:33.650 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.650+0000 7fc7baffd640 1 -- 172.21.15.83:0/553216920 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3459576673 0 0) 0x7fc7c8137b10 con 0x7fc7c814be90 2024-03-25T19:30:33.650 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.650+0000 7fc7baffd640 10 monclient(hunting): _negotiate 2024-03-25T19:30:33.650 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.650+0000 7fc7baffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:33.650 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.650+0000 7fc7baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:33.650 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.650+0000 7fc7baffd640 10 monclient(hunting): my global_id is 4729 2024-03-25T19:30:33.650 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.650+0000 7fc7baffd640 1 -- 172.21.15.83:0/553216920 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7c8138360 con 0x7fc7c814be90 2024-03-25T19:30:33.657 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.657+0000 7fc7baffd640 1 -- 172.21.15.83:0/553216920 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1900840162 0 0) 0x7fc7ac004eb0 con 0x7fc7c80b1730 2024-03-25T19:30:33.657 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.657+0000 7fc7baffd640 1 -- 172.21.15.83:0/553216920 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7c8137b10 con 0x7fc7c80b1730 2024-03-25T19:30:33.657 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.657+0000 7fc7baffd640 1 -- 172.21.15.83:0/553216920 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fc7c4001a30 con 0x7fc7c80b1730 2024-03-25T19:30:33.657 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.657+0000 7fc7baffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:33.657 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.657+0000 7fc7baffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:30:33.657 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.657+0000 7fc7baffd640 10 monclient(hunting): dump: 2024-03-25T19:30:33.657 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:30:33.657 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:30:33.657 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:33.657 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:33.657 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:33.657 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:30:33.657 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:30:33.657 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:30:33.657 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:30:33.658 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:33.658 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.657+0000 7fc7baffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:33.668 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.669+0000 7fc7baffd640 1 -- 172.21.15.83:0/553216920 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1691219443 0 0) 0x7fc7c8140980 con 0x7fc7c80b4d60 2024-03-25T19:30:33.669 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.669+0000 7fc7baffd640 1 -- 172.21.15.83:0/553216920 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7ac004eb0 con 0x7fc7c80b4d60 2024-03-25T19:30:33.669 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.669+0000 7fc7baffd640 1 -- 172.21.15.83:0/553216920 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fc7bc001b70 con 0x7fc7c80b4d60 2024-03-25T19:30:33.669 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.669+0000 7fc7baffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:33.669 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.669+0000 7fc7baffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:33.669 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.669+0000 7fc7baffd640 10 monclient(hunting): dump: 2024-03-25T19:30:33.669 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:30:33.669 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:30:33.669 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:33.669 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:33.669 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:33.669 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:30:33.669 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:30:33.669 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:30:33.669 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:30:33.669 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:33.672 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.672+0000 7fc7baffd640 1 -- 172.21.15.83:0/553216920 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 144848006 0 0) 0x7fc7c8138360 con 0x7fc7c814be90 2024-03-25T19:30:33.672 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.672+0000 7fc7baffd640 1 -- 172.21.15.83:0/553216920 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7c8140980 con 0x7fc7c814be90 2024-03-25T19:30:33.672 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.672+0000 7fc7baffd640 1 -- 172.21.15.83:0/553216920 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fc7c0001a10 con 0x7fc7c814be90 2024-03-25T19:30:33.672 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.672+0000 7fc7baffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:33.672 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.672+0000 7fc7baffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:33.672 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.672+0000 7fc7baffd640 10 monclient(hunting): dump: 2024-03-25T19:30:33.672 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:30:33.672 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:30:33.672 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:33.672 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:33.672 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:33.672 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:30:33.672 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:30:33.672 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:30:33.672 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:30:33.673 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:33.689 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.689+0000 7fc7baffd640 1 -- 172.21.15.83:0/553216920 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3412473123 0 0) 0x7fc7c8137b10 con 0x7fc7c80b1730 2024-03-25T19:30:33.689 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.689+0000 7fc7baffd640 1 -- 172.21.15.83:0/553216920 >> v1:172.21.15.83:6790/0 conn(0x7fc7c80b4d60 legacy=0x7fc7c81dab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:33.689 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.689+0000 7fc7baffd640 1 -- 172.21.15.83:0/553216920 >> v1:172.21.15.103:6789/0 conn(0x7fc7c814be90 legacy=0x7fc7c81e2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:33.689 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.689+0000 7fc7baffd640 10 monclient: _finish_hunting 0 2024-03-25T19:30:33.689 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.689+0000 7fc7baffd640 1 monclient: found mon.a 2024-03-25T19:30:33.689 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.689+0000 7fc7baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:33.689 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.689+0000 7fc7baffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:30:33.690 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.689+0000 7fc7baffd640 1 -- 172.21.15.83:0/553216920 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7c8132520 con 0x7fc7c80b1730 2024-03-25T19:30:33.690 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.689+0000 7fc7baffd640 10 monclient: _finish_auth 0 2024-03-25T19:30:33.690 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.689+0000 7fc7baffd640 10 monclient: _check_auth_tickets 2024-03-25T19:30:33.690 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.689+0000 7fc7baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:33.690 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.689+0000 7fc7d02cf640 5 monclient: authenticate success, global_id 4713 2024-03-25T19:30:33.690 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.689+0000 7fc7d02cf640 10 monclient: _renew_subs 2024-03-25T19:30:33.690 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.689+0000 7fc7d02cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:30:33.690 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.689+0000 7fc7d02cf640 1 -- 172.21.15.83:0/553216920 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7c81327d0 con 0x7fc7c80b1730 2024-03-25T19:30:33.690 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.689+0000 7fc7d02cf640 10 client.4713.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:33.690 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.689+0000 7fc7d02cf640 10 monclient: _renew_subs 2024-03-25T19:30:33.690 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.689+0000 7fc7d02cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:30:33.690 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.689+0000 7fc7d02cf640 1 -- 172.21.15.83:0/553216920 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7c8140560 con 0x7fc7c80b1730 2024-03-25T19:30:33.690 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.689+0000 7fc7d02cf640 1 librados: init done 2024-03-25T19:30:33.691 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.690+0000 7fc7d02cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:33.691 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.690+0000 7fc7d02cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:33.691 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.690+0000 7fc7d02cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:30:33.691 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:33.690+0000 7fc7d02cf640 1 -- 172.21.15.83:0/553216920 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7c80627b0 con 0x7fc7c80b1730 2024-03-25T19:30:34.038 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:34.039+0000 7fc7baffd640 1 -- 172.21.15.83:0/553216920 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc7c4001c70 con 0x7fc7c80b1730 2024-03-25T19:30:34.038 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:34.039+0000 7fc7baffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:34.038 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:34.039+0000 7fc7baffd640 1 -- 172.21.15.83:0/553216920 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fc7c4003a60 con 0x7fc7c80b1730 2024-03-25T19:30:34.039 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:34.039+0000 7fc7baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:34.039 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:34.039+0000 7fc7baffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:34.039 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:34.039+0000 7fc7baffd640 10 monclient: dump: 2024-03-25T19:30:34.039 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:30:34.039 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:30:34.039 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:34.039 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:34.039 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:34.039 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:30:34.039 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:30:34.039 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:30:34.039 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:30:34.039 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:34.039 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:34.039+0000 7fc7cc868640 4 set_mon_vals no callback set 2024-03-25T19:30:34.040 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:34.039+0000 7fc7baffd640 1 -- 172.21.15.83:0/553216920 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7fc7c4003c80 con 0x7fc7c80b1730 2024-03-25T19:30:34.040 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:34.040+0000 7fc7cd86a640 10 monclient: get_auth_request con 0x7fc7ac06e0f0 auth_method 0 2024-03-25T19:30:34.040 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:34.040+0000 7fc7baffd640 10 client.4713.objecter ms_handle_connect 0x7fc7ac06e0f0 2024-03-25T19:30:34.046 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:34.046+0000 7fc7baffd640 1 -- 172.21.15.83:0/553216920 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2659774754 0 0) 0x7fc7c4054810 con 0x7fc7c80b1730 2024-03-25T19:30:34.046 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:34.046+0000 7fc7baffd640 10 client.4713.objecter ms_dispatch 0x7fc7c8000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:30:34.046 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:34.046+0000 7fc7baffd640 3 client.4713.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:30:34.046 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:34.046+0000 7fc7baffd640 3 client.4713.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:30:34.046 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:34.046+0000 7fc7baffd640 20 client.4713.objecter dump_active .. 0 homeless 2024-03-25T19:30:34.046 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:34.046+0000 7fc7baffd640 1 -- 172.21.15.83:0/553216920 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc7c4050320 con 0x7fc7c80b1730 2024-03-25T19:30:34.047 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:34.046+0000 7fc7baffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:34.047 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:34.046+0000 7fc7baffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:34.295 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:34.295+0000 7fc7d02cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:34.295 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:34.295+0000 7fc7d02cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:34.295 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:34.295+0000 7fc7d02cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:30:34.295 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:34.295+0000 7fc7d02cf640 1 -- 172.21.15.83:0/553216920 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc7c8140b10 con 0x7fc7c80b1730 2024-03-25T19:30:34.327 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:34.327+0000 7fc7baffd640 1 -- 172.21.15.83:0/553216920 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1070525753) 0x7fc7c4052250 con 0x7fc7c80b1730 2024-03-25T19:30:34.327 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:34.327+0000 7fc7baffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:34.327 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:34.327+0000 7fc7baffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:34.327 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:30:34.327 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":11,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:29:48.933916+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:48.933916+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:34.330 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:34.330+0000 7fc7d02cf640 10 librados: watch_flush enter 2024-03-25T19:30:34.330 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:34.330+0000 7fc7d02cf640 10 librados: watch_flush exit 2024-03-25T19:30:34.330 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:34.330+0000 7fc7d02cf640 20 client.4713.objecter shutdown clearing up homeless session... 2024-03-25T19:30:34.330 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:34.330+0000 7fc7d02cf640 10 client.4713.objecter successfully canceled tick 2024-03-25T19:30:34.330 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:34.330+0000 7fc7d02cf640 1 -- 172.21.15.83:0/553216920 >> v1:172.21.15.103:6800/345072327 conn(0x7fc7ac06e0f0 legacy=0x7fc7ac0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:34.330 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:34.330+0000 7fc7d02cf640 10 monclient: shutdown 2024-03-25T19:30:34.330 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:34.330+0000 7fc7d02cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:34.331 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:34.330+0000 7fc7d02cf640 1 -- 172.21.15.83:0/553216920 >> v1:172.21.15.83:6789/0 conn(0x7fc7c80b1730 legacy=0x7fc7c8131e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:34.331 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:34.330+0000 7fc7d02cf640 1 -- 172.21.15.83:0/553216920 shutdown_connections 2024-03-25T19:30:34.331 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:34.330+0000 7fc7d02cf640 1 -- 172.21.15.83:0/553216920 >> 172.21.15.83:0/553216920 conn(0x7fc7c80617a0 msgr2=0x7fc7c80b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:34.331 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:34.330+0000 7fc7d02cf640 1 -- 172.21.15.83:0/553216920 shutdown_connections 2024-03-25T19:30:34.331 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:34.330+0000 7fc7d02cf640 1 -- 172.21.15.83:0/553216920 wait complete. 2024-03-25T19:30:34.331 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:34.330+0000 7fc7d02cf640 1 librados: shutdown 2024-03-25T19:30:34.341 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:41.147 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:41.252 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.252+0000 7ffaedf32640 10 monclient: get_monmap_and_config 2024-03-25T19:30:41.252 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.252+0000 7ffaedf32640 10 monclient: build_initial_monmap 2024-03-25T19:30:41.252 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.252+0000 7ffaedf32640 10 monclient: monmap: 2024-03-25T19:30:41.252 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:30:41.252 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:30:41.252 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:30:41.253488+0000 2024-03-25T19:30:41.252 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:30:41.253488+0000 2024-03-25T19:30:41.252 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:41.252 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:30:41.252 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:30:41.252 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:30:41.252 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:30:41.253 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:41.254 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.253+0000 7ffaedf32640 1 Processor -- start 2024-03-25T19:30:41.254 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.253+0000 7ffaedf32640 1 -- start start 2024-03-25T19:30:41.254 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.253+0000 7ffaedf32640 10 monclient: init 2024-03-25T19:30:41.254 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.253+0000 7ffaedf32640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:41.254 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.253+0000 7ffaedf32640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:30:41.254 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.253+0000 7ffaedf32640 10 monclient(hunting): picked mon.noname-b con 0x7ffae80b4d60 addr v1:172.21.15.83:6790/0 2024-03-25T19:30:41.254 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.253+0000 7ffaedf32640 10 monclient(hunting): picked mon.noname-a con 0x7ffae80b1730 addr v1:172.21.15.83:6789/0 2024-03-25T19:30:41.254 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.253+0000 7ffaedf32640 10 monclient(hunting): picked mon.noname-c con 0x7ffae814be90 addr v1:172.21.15.103:6789/0 2024-03-25T19:30:41.254 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.253+0000 7ffaedf32640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffae80af940 con 0x7ffae80b1730 2024-03-25T19:30:41.255 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.253+0000 7ffaedf32640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffae80b0a70 con 0x7ffae814be90 2024-03-25T19:30:41.255 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.253+0000 7ffaedf32640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffae80b0d50 con 0x7ffae80b4d60 2024-03-25T19:30:41.255 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.253+0000 7ffaedf32640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:41.255 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.253+0000 7ffaedf32640 10 monclient(hunting): authenticate will time out at 6698.525724s 2024-03-25T19:30:41.255 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.254+0000 7ffae77fe640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7ffae80b4d60 0x7ffae813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:37658/0 (socket says 172.21.15.83:37658) 2024-03-25T19:30:41.255 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.254+0000 7ffae77fe640 1 -- 172.21.15.83:0/3217457190 learned_addr learned my addr 172.21.15.83:0/3217457190 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:30:41.295 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.296+0000 7ffae67fc640 1 -- 172.21.15.83:0/3217457190 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3950522982 0 0) 0x7ffae80af940 con 0x7ffae80b1730 2024-03-25T19:30:41.295 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.296+0000 7ffae67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:30:41.295 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.296+0000 7ffae67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:41.295 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.296+0000 7ffae67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:41.295 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.296+0000 7ffae67fc640 10 monclient(hunting): my global_id is 4716 2024-03-25T19:30:41.296 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.296+0000 7ffae67fc640 1 -- 172.21.15.83:0/3217457190 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffad0003520 con 0x7ffae80b1730 2024-03-25T19:30:41.319 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.319+0000 7ffae67fc640 1 -- 172.21.15.83:0/3217457190 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1132540109 0 0) 0x7ffad0003520 con 0x7ffae80b1730 2024-03-25T19:30:41.319 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.319+0000 7ffae67fc640 1 -- 172.21.15.83:0/3217457190 >> v1:172.21.15.83:6790/0 conn(0x7ffae80b4d60 legacy=0x7ffae813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:41.319 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.319+0000 7ffae67fc640 1 -- 172.21.15.83:0/3217457190 >> v1:172.21.15.103:6789/0 conn(0x7ffae814be90 legacy=0x7ffae814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:41.319 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.319+0000 7ffae67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:30:41.319 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.319+0000 7ffae67fc640 1 monclient: found mon.noname-a 2024-03-25T19:30:41.319 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.319+0000 7ffae67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:41.319 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.319+0000 7ffae67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.83:6789/0 2024-03-25T19:30:41.319 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.319+0000 7ffae67fc640 1 -- 172.21.15.83:0/3217457190 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae80b1030 con 0x7ffae80b1730 2024-03-25T19:30:41.319 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.319+0000 7ffae67fc640 10 monclient: _finish_auth 0 2024-03-25T19:30:41.320 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.319+0000 7ffae67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:30:41.320 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.319+0000 7ffae67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:41.320 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.319+0000 7ffae67fc640 1 -- 172.21.15.83:0/3217457190 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7ffadc001690 con 0x7ffae80b1730 2024-03-25T19:30:41.320 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.319+0000 7ffae67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:41.320 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.319+0000 7ffae67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:30:41.320 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.319+0000 7ffae67fc640 10 monclient: dump: 2024-03-25T19:30:41.320 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:30:41.320 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:30:41.320 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:41.320 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:41.320 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:41.320 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:30:41.320 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:30:41.320 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:30:41.320 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:30:41.320 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:41.321 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.319+0000 7ffaedf32640 5 monclient: authenticate success, global_id 4716 2024-03-25T19:30:41.321 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.319+0000 7ffaedf32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:41.346 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.347+0000 7ffae67fc640 1 -- 172.21.15.83:0/3217457190 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffadc0017f0 con 0x7ffae80b1730 2024-03-25T19:30:41.347 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.347+0000 7ffae67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:41.347 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.347+0000 7ffae67fc640 1 -- 172.21.15.83:0/3217457190 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7ffadc001c00 con 0x7ffae80b1730 2024-03-25T19:30:41.347 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.347+0000 7ffaedf32640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:41.347 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.347+0000 7ffaedf32640 4 set_mon_vals no callback set 2024-03-25T19:30:41.347 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.347+0000 7ffae67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:41.347 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.347+0000 7ffae67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:41.347 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.347+0000 7ffae67fc640 10 monclient: dump: 2024-03-25T19:30:41.347 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:30:41.347 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:30:41.347 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:41.347 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:41.347 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:41.347 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:30:41.347 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:30:41.347 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:30:41.348 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:30:41.348 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:41.350 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.347+0000 7ffaedf32640 10 monclient: shutdown 2024-03-25T19:30:41.350 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.347+0000 7ffaedf32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:41.350 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.347+0000 7ffaedf32640 1 -- 172.21.15.83:0/3217457190 >> v1:172.21.15.83:6789/0 conn(0x7ffae80b1730 legacy=0x7ffae80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:41.350 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.348+0000 7ffaedf32640 1 -- 172.21.15.83:0/3217457190 shutdown_connections 2024-03-25T19:30:41.350 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.348+0000 7ffaedf32640 1 -- 172.21.15.83:0/3217457190 >> 172.21.15.83:0/3217457190 conn(0x7ffae80617a0 msgr2=0x7ffae8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:41.350 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.348+0000 7ffaedf32640 1 -- 172.21.15.83:0/3217457190 shutdown_connections 2024-03-25T19:30:41.351 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.348+0000 7ffaedf32640 1 -- 172.21.15.83:0/3217457190 wait complete. 2024-03-25T19:30:41.351 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.348+0000 7ffaedf32640 10 monclient: build_initial_monmap 2024-03-25T19:30:41.351 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.348+0000 7ffaedf32640 10 monclient: monmap: 2024-03-25T19:30:41.351 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:30:41.351 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:41.351 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:30:41.351 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:30:41.351 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:41.351 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:30:41.351 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:30:41.351 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:30:41.351 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:30:41.351 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:41.351 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.349+0000 7ffaedf32640 1 librados: starting msgr at 2024-03-25T19:30:41.351 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.349+0000 7ffaedf32640 1 librados: starting objecter 2024-03-25T19:30:41.351 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.349+0000 7ffaedf32640 1 Processor -- start 2024-03-25T19:30:41.352 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.349+0000 7ffaedf32640 1 -- start start 2024-03-25T19:30:41.352 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.349+0000 7ffaedf32640 1 librados: setting wanted keys 2024-03-25T19:30:41.352 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.349+0000 7ffaedf32640 1 librados: calling monclient init 2024-03-25T19:30:41.352 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.349+0000 7ffaedf32640 10 monclient: init 2024-03-25T19:30:41.352 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.350+0000 7ffaedf32640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:41.352 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.350+0000 7ffaedf32640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:30:41.352 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.350+0000 7ffaedf32640 10 monclient(hunting): picked mon.noname-a con 0x7ffae80b1730 addr v1:172.21.15.83:6789/0 2024-03-25T19:30:41.352 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.350+0000 7ffaedf32640 10 monclient(hunting): picked mon.noname-c con 0x7ffae80b4d60 addr v1:172.21.15.83:6790/0 2024-03-25T19:30:41.352 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.350+0000 7ffaedf32640 10 monclient(hunting): picked mon.noname-b con 0x7ffae814be90 addr v1:172.21.15.103:6789/0 2024-03-25T19:30:41.352 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.350+0000 7ffaedf32640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffae8143690 con 0x7ffae80b1730 2024-03-25T19:30:41.352 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.350+0000 7ffaedf32640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffae813ee90 con 0x7ffae814be90 2024-03-25T19:30:41.352 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.350+0000 7ffaedf32640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffae813f020 con 0x7ffae80b4d60 2024-03-25T19:30:41.352 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.350+0000 7ffaedf32640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:41.352 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.350+0000 7ffaedf32640 10 monclient(hunting): authenticate will time out at 6698.621916s 2024-03-25T19:30:41.357 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.358+0000 7ffae6ffd640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7ffae80b4d60 0x7ffae81d7a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:37706/0 (socket says 172.21.15.83:37706) 2024-03-25T19:30:41.358 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.358+0000 7ffae6ffd640 1 -- 172.21.15.83:0/1404097103 learned_addr learned my addr 172.21.15.83:0/1404097103 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:30:41.358 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.358+0000 7ffacffff640 10 client.?.objecter ms_handle_connect 0x7ffae80b4d60 2024-03-25T19:30:41.358 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.358+0000 7ffacffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:41.368 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.368+0000 7ffacffff640 1 -- 172.21.15.83:0/1404097103 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1409998161 0 0) 0x7ffae813f020 con 0x7ffae80b4d60 2024-03-25T19:30:41.368 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.368+0000 7ffacffff640 10 monclient(hunting): _negotiate 2024-03-25T19:30:41.368 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.368+0000 7ffacffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:41.368 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.368+0000 7ffacffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:41.368 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.368+0000 7ffacffff640 10 monclient(hunting): my global_id is 4730 2024-03-25T19:30:41.368 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.368+0000 7ffacffff640 1 -- 172.21.15.83:0/1404097103 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffad4004eb0 con 0x7ffae80b4d60 2024-03-25T19:30:41.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.371+0000 7ffacffff640 10 client.?.objecter ms_handle_connect 0x7ffae814be90 2024-03-25T19:30:41.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.371+0000 7ffacffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:41.380 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.380+0000 7ffacffff640 1 -- 172.21.15.83:0/1404097103 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 888619020 0 0) 0x7ffae813ee90 con 0x7ffae814be90 2024-03-25T19:30:41.380 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.380+0000 7ffacffff640 10 monclient(hunting): _negotiate 2024-03-25T19:30:41.380 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.380+0000 7ffacffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:41.380 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.380+0000 7ffacffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:41.380 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.380+0000 7ffacffff640 10 monclient(hunting): my global_id is 4735 2024-03-25T19:30:41.380 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.380+0000 7ffacffff640 1 -- 172.21.15.83:0/1404097103 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffae813f020 con 0x7ffae814be90 2024-03-25T19:30:41.392 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.392+0000 7ffacffff640 1 -- 172.21.15.83:0/1404097103 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1431562979 0 0) 0x7ffad4004eb0 con 0x7ffae80b4d60 2024-03-25T19:30:41.392 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.392+0000 7ffacffff640 1 -- 172.21.15.83:0/1404097103 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffae813ee90 con 0x7ffae80b4d60 2024-03-25T19:30:41.392 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.392+0000 7ffacffff640 1 -- 172.21.15.83:0/1404097103 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7ffadc001690 con 0x7ffae80b4d60 2024-03-25T19:30:41.392 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.392+0000 7ffacffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:41.393 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.392+0000 7ffacffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:30:41.393 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.392+0000 7ffacffff640 10 monclient(hunting): dump: 2024-03-25T19:30:41.393 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:30:41.393 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:30:41.393 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:41.393 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:41.393 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:41.393 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:30:41.393 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:30:41.393 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:30:41.393 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:30:41.393 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:41.393 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.392+0000 7ffacffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:41.396 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.397+0000 7ffacffff640 10 client.?.objecter ms_handle_connect 0x7ffae80b1730 2024-03-25T19:30:41.397 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.397+0000 7ffacffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:41.410 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.410+0000 7ffacffff640 1 -- 172.21.15.83:0/1404097103 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1844064238 0 0) 0x7ffae8143690 con 0x7ffae80b1730 2024-03-25T19:30:41.410 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.410+0000 7ffacffff640 10 monclient(hunting): _negotiate 2024-03-25T19:30:41.410 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.410+0000 7ffacffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:41.410 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.410+0000 7ffacffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:41.410 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.410+0000 7ffacffff640 10 monclient(hunting): my global_id is 4719 2024-03-25T19:30:41.410 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.410+0000 7ffacffff640 1 -- 172.21.15.83:0/1404097103 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffad4004eb0 con 0x7ffae80b1730 2024-03-25T19:30:41.410 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.410+0000 7ffacffff640 1 -- 172.21.15.83:0/1404097103 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2175212121 0 0) 0x7ffae813f020 con 0x7ffae814be90 2024-03-25T19:30:41.410 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.410+0000 7ffacffff640 1 -- 172.21.15.83:0/1404097103 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffae8143690 con 0x7ffae814be90 2024-03-25T19:30:41.410 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.410+0000 7ffacffff640 1 -- 172.21.15.83:0/1404097103 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7ffae0001b00 con 0x7ffae814be90 2024-03-25T19:30:41.410 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.410+0000 7ffacffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:41.411 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.410+0000 7ffacffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:41.411 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.410+0000 7ffacffff640 10 monclient(hunting): dump: 2024-03-25T19:30:41.411 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:30:41.411 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:30:41.411 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:41.411 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:41.411 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:41.411 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:30:41.411 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:30:41.411 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:30:41.411 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:30:41.411 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:41.428 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.427+0000 7ffacffff640 1 -- 172.21.15.83:0/1404097103 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2082554854 0 0) 0x7ffae813ee90 con 0x7ffae80b4d60 2024-03-25T19:30:41.428 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.428+0000 7ffacffff640 1 -- 172.21.15.83:0/1404097103 >> v1:172.21.15.103:6789/0 conn(0x7ffae814be90 legacy=0x7ffae81d8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:41.428 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.428+0000 7ffacffff640 1 -- 172.21.15.83:0/1404097103 >> v1:172.21.15.83:6789/0 conn(0x7ffae80b1730 legacy=0x7ffae813b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:41.428 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.428+0000 7ffacffff640 10 monclient: _finish_hunting 0 2024-03-25T19:30:41.428 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.428+0000 7ffacffff640 1 monclient: found mon.c 2024-03-25T19:30:41.428 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.428+0000 7ffacffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:41.428 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.428+0000 7ffacffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:30:41.428 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.428+0000 7ffacffff640 1 -- 172.21.15.83:0/1404097103 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae813bb30 con 0x7ffae80b4d60 2024-03-25T19:30:41.428 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.428+0000 7ffacffff640 10 monclient: _finish_auth 0 2024-03-25T19:30:41.428 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.428+0000 7ffacffff640 10 monclient: _check_auth_tickets 2024-03-25T19:30:41.428 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.428+0000 7ffacffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:41.428 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.428+0000 7ffaedf32640 5 monclient: authenticate success, global_id 4730 2024-03-25T19:30:41.428 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.428+0000 7ffaedf32640 10 monclient: _renew_subs 2024-03-25T19:30:41.428 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.428+0000 7ffaedf32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:30:41.429 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.428+0000 7ffaedf32640 1 -- 172.21.15.83:0/1404097103 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffae813bd80 con 0x7ffae80b4d60 2024-03-25T19:30:41.429 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.428+0000 7ffaedf32640 10 client.4730.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:41.429 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.428+0000 7ffaedf32640 10 monclient: _renew_subs 2024-03-25T19:30:41.429 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.428+0000 7ffaedf32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:30:41.429 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.428+0000 7ffaedf32640 1 -- 172.21.15.83:0/1404097103 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffae8143f10 con 0x7ffae80b4d60 2024-03-25T19:30:41.429 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.428+0000 7ffaedf32640 1 librados: init done 2024-03-25T19:30:41.429 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.429+0000 7ffaedf32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:41.429 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.429+0000 7ffaedf32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:41.430 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.429+0000 7ffaedf32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:30:41.430 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.429+0000 7ffaedf32640 1 -- 172.21.15.83:0/1404097103 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffae80b3ca0 con 0x7ffae80b4d60 2024-03-25T19:30:41.755 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.755+0000 7ffacffff640 1 -- 172.21.15.83:0/1404097103 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffadc0038a0 con 0x7ffae80b4d60 2024-03-25T19:30:41.755 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.755+0000 7ffacffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:41.755 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.755+0000 7ffacffff640 1 -- 172.21.15.83:0/1404097103 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7ffadc003cf0 con 0x7ffae80b4d60 2024-03-25T19:30:41.755 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.755+0000 7ffacffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:41.755 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.755+0000 7ffacffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:41.755 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.755+0000 7ffacffff640 10 monclient: dump: 2024-03-25T19:30:41.755 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:30:41.755 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:30:41.755 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:41.755 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:41.755 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:41.755 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:30:41.756 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:30:41.756 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:30:41.756 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:30:41.756 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:41.756 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.755+0000 7ffae5ffb640 4 set_mon_vals no callback set 2024-03-25T19:30:41.756 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.755+0000 7ffacffff640 1 -- 172.21.15.83:0/1404097103 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7ffadc003f10 con 0x7ffae80b4d60 2024-03-25T19:30:41.756 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.756+0000 7ffae77fe640 10 monclient: get_auth_request con 0x7ffad406df70 auth_method 0 2024-03-25T19:30:41.756 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.756+0000 7ffacffff640 10 client.4730.objecter ms_handle_connect 0x7ffad406df70 2024-03-25T19:30:41.763 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.763+0000 7ffacffff640 1 -- 172.21.15.83:0/1404097103 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2659774754 0 0) 0x7ffadc054930 con 0x7ffae80b4d60 2024-03-25T19:30:41.763 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.763+0000 7ffacffff640 10 client.4730.objecter ms_dispatch 0x7ffae8000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:30:41.763 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.763+0000 7ffacffff640 3 client.4730.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:30:41.763 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.763+0000 7ffacffff640 3 client.4730.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:30:41.763 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.763+0000 7ffacffff640 20 client.4730.objecter dump_active .. 0 homeless 2024-03-25T19:30:41.763 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.763+0000 7ffacffff640 1 -- 172.21.15.83:0/1404097103 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffadc050440 con 0x7ffae80b4d60 2024-03-25T19:30:41.763 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.763+0000 7ffacffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:41.763 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:41.763+0000 7ffacffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:42.028 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:42.028+0000 7ffaedf32640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:42.028 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:42.028+0000 7ffaedf32640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:42.028 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:42.028+0000 7ffaedf32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:30:42.028 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:42.028+0000 7ffaedf32640 1 -- 172.21.15.83:0/1404097103 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffae80627b0 con 0x7ffae80b4d60 2024-03-25T19:30:42.066 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:42.066+0000 7ffacffff640 1 -- 172.21.15.83:0/1404097103 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1070525753) 0x7ffadc052370 con 0x7ffae80b4d60 2024-03-25T19:30:42.066 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:42.066+0000 7ffacffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:42.066 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:42.066+0000 7ffacffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:42.067 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:30:42.067 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":11,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:29:48.933916+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:48.933916+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:42.070 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:42.069+0000 7ffaedf32640 10 librados: watch_flush enter 2024-03-25T19:30:42.070 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:42.069+0000 7ffaedf32640 10 librados: watch_flush exit 2024-03-25T19:30:42.070 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:42.069+0000 7ffaedf32640 20 client.4730.objecter shutdown clearing up homeless session... 2024-03-25T19:30:42.070 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:42.069+0000 7ffaedf32640 10 client.4730.objecter successfully canceled tick 2024-03-25T19:30:42.070 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:42.069+0000 7ffaedf32640 1 -- 172.21.15.83:0/1404097103 >> v1:172.21.15.103:6800/345072327 conn(0x7ffad406df70 legacy=0x7ffad4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:42.070 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:42.069+0000 7ffaedf32640 10 monclient: shutdown 2024-03-25T19:30:42.070 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:42.069+0000 7ffaedf32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:42.070 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:42.069+0000 7ffaedf32640 1 -- 172.21.15.83:0/1404097103 >> v1:172.21.15.83:6790/0 conn(0x7ffae80b4d60 legacy=0x7ffae81d7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:42.070 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:42.069+0000 7ffaedf32640 1 -- 172.21.15.83:0/1404097103 shutdown_connections 2024-03-25T19:30:42.070 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:42.069+0000 7ffaedf32640 1 -- 172.21.15.83:0/1404097103 >> 172.21.15.83:0/1404097103 conn(0x7ffae80617a0 msgr2=0x7ffae813f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:42.070 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:42.070+0000 7ffaedf32640 1 -- 172.21.15.83:0/1404097103 shutdown_connections 2024-03-25T19:30:42.070 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:42.070+0000 7ffaedf32640 1 -- 172.21.15.83:0/1404097103 wait complete. 2024-03-25T19:30:42.070 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:42.070+0000 7ffaedf32640 1 librados: shutdown 2024-03-25T19:30:42.082 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:48.888 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:49.005 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.005+0000 7fab9d60d640 10 monclient: get_monmap_and_config 2024-03-25T19:30:49.005 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.005+0000 7fab9d60d640 10 monclient: build_initial_monmap 2024-03-25T19:30:49.005 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.005+0000 7fab9d60d640 10 monclient: monmap: 2024-03-25T19:30:49.005 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:30:49.005 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:30:49.005 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:30:49.006097+0000 2024-03-25T19:30:49.005 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:30:49.006097+0000 2024-03-25T19:30:49.005 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:49.005 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:30:49.006 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:30:49.006 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:30:49.006 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:30:49.006 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:49.007 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.006+0000 7fab9d60d640 1 Processor -- start 2024-03-25T19:30:49.007 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.006+0000 7fab9d60d640 1 -- start start 2024-03-25T19:30:49.007 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.006+0000 7fab9d60d640 10 monclient: init 2024-03-25T19:30:49.007 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.006+0000 7fab9d60d640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:49.007 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.006+0000 7fab9d60d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:30:49.007 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.006+0000 7fab9d60d640 10 monclient(hunting): picked mon.noname-b con 0x7fab980a98f0 addr v1:172.21.15.83:6790/0 2024-03-25T19:30:49.007 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.006+0000 7fab9d60d640 10 monclient(hunting): picked mon.noname-c con 0x7fab9813f710 addr v1:172.21.15.103:6789/0 2024-03-25T19:30:49.007 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.006+0000 7fab9d60d640 10 monclient(hunting): picked mon.noname-a con 0x7fab981430c0 addr v1:172.21.15.83:6789/0 2024-03-25T19:30:49.007 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.006+0000 7fab9d60d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab98142520 con 0x7fab981430c0 2024-03-25T19:30:49.007 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.006+0000 7fab9d60d640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab980b28a0 con 0x7fab9813f710 2024-03-25T19:30:49.007 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.006+0000 7fab9d60d640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab980b2a30 con 0x7fab980a98f0 2024-03-25T19:30:49.007 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.006+0000 7fab9d60d640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:49.007 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.006+0000 7fab9d60d640 10 monclient(hunting): authenticate will time out at 6706.278606s 2024-03-25T19:30:49.008 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.008+0000 7fab96ffd640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fab980a98f0 0x7fab980a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:45404/0 (socket says 172.21.15.83:45404) 2024-03-25T19:30:49.008 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.008+0000 7fab96ffd640 1 -- 172.21.15.83:0/3850337869 learned_addr learned my addr 172.21.15.83:0/3850337869 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:30:49.016 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.016+0000 7fab967fc640 1 -- 172.21.15.83:0/3850337869 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 387645505 0 0) 0x7fab980b2a30 con 0x7fab980a98f0 2024-03-25T19:30:49.016 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.016+0000 7fab967fc640 10 monclient(hunting): _negotiate 2024-03-25T19:30:49.016 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.016+0000 7fab967fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:49.016 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.016+0000 7fab967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:49.016 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.016+0000 7fab967fc640 10 monclient(hunting): my global_id is 4733 2024-03-25T19:30:49.016 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.016+0000 7fab967fc640 1 -- 172.21.15.83:0/3850337869 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab84003520 con 0x7fab980a98f0 2024-03-25T19:30:49.024 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.024+0000 7fab967fc640 1 -- 172.21.15.83:0/3850337869 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3419166269 0 0) 0x7fab84003520 con 0x7fab980a98f0 2024-03-25T19:30:49.024 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.024+0000 7fab967fc640 1 -- 172.21.15.83:0/3850337869 >> v1:172.21.15.103:6789/0 conn(0x7fab9813f710 legacy=0x7fab981429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:30:49.024 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.024+0000 7fab967fc640 1 -- 172.21.15.83:0/3850337869 >> v1:172.21.15.83:6789/0 conn(0x7fab981430c0 legacy=0x7fab9814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:49.024 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.024+0000 7fab967fc640 10 monclient: _finish_hunting 0 2024-03-25T19:30:49.024 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.024+0000 7fab967fc640 1 monclient: found mon.noname-b 2024-03-25T19:30:49.025 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.024+0000 7fab967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:49.025 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.024+0000 7fab967fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.83:6790/0 2024-03-25T19:30:49.025 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.024+0000 7fab967fc640 1 -- 172.21.15.83:0/3850337869 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab980b44c0 con 0x7fab980a98f0 2024-03-25T19:30:49.025 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.024+0000 7fab967fc640 10 monclient: _finish_auth 0 2024-03-25T19:30:49.025 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.024+0000 7fab967fc640 10 monclient: _check_auth_tickets 2024-03-25T19:30:49.025 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.024+0000 7fab967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:49.025 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.024+0000 7fab9d60d640 5 monclient: authenticate success, global_id 4733 2024-03-25T19:30:49.025 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.024+0000 7fab967fc640 1 -- 172.21.15.83:0/3850337869 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fab88001690 con 0x7fab980a98f0 2024-03-25T19:30:49.025 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.024+0000 7fab967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:49.025 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.024+0000 7fab967fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:30:49.025 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.024+0000 7fab967fc640 10 monclient: dump: 2024-03-25T19:30:49.025 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:30:49.025 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:30:49.025 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:49.025 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:49.026 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:49.026 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:30:49.026 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:30:49.026 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:30:49.026 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:30:49.026 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:49.026 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.024+0000 7fab9d60d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:49.040 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.040+0000 7fab967fc640 1 -- 172.21.15.83:0/3850337869 <== mon.2 v1:172.21.15.83:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fab88001860 con 0x7fab980a98f0 2024-03-25T19:30:49.040 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.040+0000 7fab967fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:49.040 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.040+0000 7fab967fc640 1 -- 172.21.15.83:0/3850337869 <== mon.2 v1:172.21.15.83:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fab88001cb0 con 0x7fab980a98f0 2024-03-25T19:30:49.040 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.040+0000 7fab9d60d640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:49.040 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.040+0000 7fab9d60d640 4 set_mon_vals no callback set 2024-03-25T19:30:49.040 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.040+0000 7fab967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:49.040 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.040+0000 7fab967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:49.040 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.040+0000 7fab967fc640 10 monclient: dump: 2024-03-25T19:30:49.040 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:30:49.041 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:30:49.041 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:49.041 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:49.041 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:49.041 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:30:49.041 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:30:49.041 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:30:49.041 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:30:49.041 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:49.043 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.041+0000 7fab9d60d640 10 monclient: shutdown 2024-03-25T19:30:49.043 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.041+0000 7fab9d60d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:49.043 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.041+0000 7fab9d60d640 1 -- 172.21.15.83:0/3850337869 >> v1:172.21.15.83:6790/0 conn(0x7fab980a98f0 legacy=0x7fab980a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:49.043 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.041+0000 7fab9d60d640 1 -- 172.21.15.83:0/3850337869 shutdown_connections 2024-03-25T19:30:49.043 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.041+0000 7fab9d60d640 1 -- 172.21.15.83:0/3850337869 >> 172.21.15.83:0/3850337869 conn(0x7fab980b7ea0 msgr2=0x7fab980b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:49.043 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.041+0000 7fab9d60d640 1 -- 172.21.15.83:0/3850337869 shutdown_connections 2024-03-25T19:30:49.043 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.041+0000 7fab9d60d640 1 -- 172.21.15.83:0/3850337869 wait complete. 2024-03-25T19:30:49.043 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.042+0000 7fab9d60d640 10 monclient: build_initial_monmap 2024-03-25T19:30:49.043 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.042+0000 7fab9d60d640 10 monclient: monmap: 2024-03-25T19:30:49.043 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:30:49.043 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:49.043 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:30:49.044 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:30:49.044 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:49.044 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:30:49.044 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:30:49.044 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:30:49.044 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:30:49.044 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:49.044 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.042+0000 7fab9d60d640 1 librados: starting msgr at 2024-03-25T19:30:49.044 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.043+0000 7fab9d60d640 1 librados: starting objecter 2024-03-25T19:30:49.044 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.043+0000 7fab9d60d640 1 Processor -- start 2024-03-25T19:30:49.044 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.043+0000 7fab9d60d640 1 -- start start 2024-03-25T19:30:49.044 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.043+0000 7fab9d60d640 1 librados: setting wanted keys 2024-03-25T19:30:49.044 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.043+0000 7fab9d60d640 1 librados: calling monclient init 2024-03-25T19:30:49.044 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.043+0000 7fab9d60d640 10 monclient: init 2024-03-25T19:30:49.045 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.043+0000 7fab9d60d640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:49.045 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.043+0000 7fab9d60d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:30:49.046 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.043+0000 7fab9d60d640 10 monclient(hunting): picked mon.noname-a con 0x7fab980a98f0 addr v1:172.21.15.83:6789/0 2024-03-25T19:30:49.046 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.043+0000 7fab9d60d640 10 monclient(hunting): picked mon.noname-b con 0x7fab9813f710 addr v1:172.21.15.103:6789/0 2024-03-25T19:30:49.046 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.044+0000 7fab9d60d640 10 monclient(hunting): picked mon.noname-c con 0x7fab981430c0 addr v1:172.21.15.83:6790/0 2024-03-25T19:30:49.046 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.044+0000 7fab9d60d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab981df1d0 con 0x7fab980a98f0 2024-03-25T19:30:49.046 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.044+0000 7fab9d60d640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab980708d0 con 0x7fab9813f710 2024-03-25T19:30:49.046 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.044+0000 7fab9d60d640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab98070cf0 con 0x7fab981430c0 2024-03-25T19:30:49.046 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.044+0000 7fab9d60d640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:49.046 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.044+0000 7fab9d60d640 10 monclient(hunting): authenticate will time out at 6706.315864s 2024-03-25T19:30:49.054 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.054+0000 7fab8ffff640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fab9813f710 0x7fab9806ebc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:40944/0 (socket says 172.21.15.83:40944) 2024-03-25T19:30:49.054 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.054+0000 7fab8ffff640 1 -- 172.21.15.83:0/99456225 learned_addr learned my addr 172.21.15.83:0/99456225 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:30:49.059 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.059+0000 7fab8f7fe640 10 client.?.objecter ms_handle_connect 0x7fab980a98f0 2024-03-25T19:30:49.059 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.059+0000 7fab8f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:49.060 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.060+0000 7fab8f7fe640 10 client.?.objecter ms_handle_connect 0x7fab9813f710 2024-03-25T19:30:49.060 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.060+0000 7fab8f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:49.061 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.061+0000 7fab8f7fe640 10 client.?.objecter ms_handle_connect 0x7fab981430c0 2024-03-25T19:30:49.061 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.061+0000 7fab8f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:49.070 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.070+0000 7fab8f7fe640 1 -- 172.21.15.83:0/99456225 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1137119028 0 0) 0x7fab981df1d0 con 0x7fab980a98f0 2024-03-25T19:30:49.070 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.070+0000 7fab8f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:30:49.070 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.070+0000 7fab8f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:49.070 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.070+0000 7fab8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:49.071 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.070+0000 7fab8f7fe640 10 monclient(hunting): my global_id is 4722 2024-03-25T19:30:49.071 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.070+0000 7fab8f7fe640 1 -- 172.21.15.83:0/99456225 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab78004eb0 con 0x7fab980a98f0 2024-03-25T19:30:49.074 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.074+0000 7fab8f7fe640 1 -- 172.21.15.83:0/99456225 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 133675315 0 0) 0x7fab980708d0 con 0x7fab9813f710 2024-03-25T19:30:49.074 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.074+0000 7fab8f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:30:49.074 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.074+0000 7fab8f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:49.074 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.074+0000 7fab8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:49.074 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.074+0000 7fab8f7fe640 10 monclient(hunting): my global_id is 4738 2024-03-25T19:30:49.074 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.074+0000 7fab8f7fe640 1 -- 172.21.15.83:0/99456225 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab981df1d0 con 0x7fab9813f710 2024-03-25T19:30:49.077 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.077+0000 7fab8f7fe640 1 -- 172.21.15.83:0/99456225 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1324143650 0 0) 0x7fab98070cf0 con 0x7fab981430c0 2024-03-25T19:30:49.077 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.077+0000 7fab8f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:30:49.077 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.077+0000 7fab8f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:49.077 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.077+0000 7fab8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:49.077 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.077+0000 7fab8f7fe640 10 monclient(hunting): my global_id is 4736 2024-03-25T19:30:49.077 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.077+0000 7fab8f7fe640 1 -- 172.21.15.83:0/99456225 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab980708d0 con 0x7fab981430c0 2024-03-25T19:30:49.079 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.080+0000 7fab8f7fe640 1 -- 172.21.15.83:0/99456225 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1414914229 0 0) 0x7fab78004eb0 con 0x7fab980a98f0 2024-03-25T19:30:49.080 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.080+0000 7fab8f7fe640 1 -- 172.21.15.83:0/99456225 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab98070cf0 con 0x7fab980a98f0 2024-03-25T19:30:49.082 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.082+0000 7fab8f7fe640 1 -- 172.21.15.83:0/99456225 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fab88001b70 con 0x7fab980a98f0 2024-03-25T19:30:49.082 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.082+0000 7fab8f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:49.082 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.082+0000 7fab8f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:30:49.083 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.082+0000 7fab8f7fe640 10 monclient(hunting): dump: 2024-03-25T19:30:49.083 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:30:49.083 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:30:49.083 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:49.083 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:49.083 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:49.083 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:30:49.083 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:30:49.083 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:30:49.083 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:30:49.083 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:49.083 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.082+0000 7fab8f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:49.092 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.091+0000 7fab8f7fe640 1 -- 172.21.15.83:0/99456225 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2815035264 0 0) 0x7fab98070cf0 con 0x7fab980a98f0 2024-03-25T19:30:49.092 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.091+0000 7fab8f7fe640 1 -- 172.21.15.83:0/99456225 >> v1:172.21.15.83:6790/0 conn(0x7fab981430c0 legacy=0x7fab9806f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:49.092 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.091+0000 7fab8f7fe640 1 -- 172.21.15.83:0/99456225 >> v1:172.21.15.103:6789/0 conn(0x7fab9813f710 legacy=0x7fab9806ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:49.092 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.091+0000 7fab8f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:30:49.092 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.091+0000 7fab8f7fe640 1 monclient: found mon.a 2024-03-25T19:30:49.092 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.091+0000 7fab8f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:49.092 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.091+0000 7fab8f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:30:49.092 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.091+0000 7fab8f7fe640 1 -- 172.21.15.83:0/99456225 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab98070e80 con 0x7fab980a98f0 2024-03-25T19:30:49.092 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.091+0000 7fab8f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:30:49.092 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.091+0000 7fab8f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:30:49.093 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.091+0000 7fab8f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:49.093 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.091+0000 7fab9d60d640 5 monclient: authenticate success, global_id 4722 2024-03-25T19:30:49.093 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.091+0000 7fab9d60d640 10 monclient: _renew_subs 2024-03-25T19:30:49.093 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.091+0000 7fab9d60d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:30:49.093 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.091+0000 7fab9d60d640 1 -- 172.21.15.83:0/99456225 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab981dfdc0 con 0x7fab980a98f0 2024-03-25T19:30:49.093 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.091+0000 7fab9d60d640 10 client.4722.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:49.093 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.091+0000 7fab9d60d640 10 monclient: _renew_subs 2024-03-25T19:30:49.093 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.091+0000 7fab9d60d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:30:49.093 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.091+0000 7fab9d60d640 1 -- 172.21.15.83:0/99456225 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fab981e78c0 con 0x7fab980a98f0 2024-03-25T19:30:49.093 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.091+0000 7fab9d60d640 1 librados: init done 2024-03-25T19:30:49.093 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.092+0000 7fab9d60d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:49.093 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.092+0000 7fab9d60d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:49.093 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.092+0000 7fab9d60d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:30:49.093 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.092+0000 7fab9d60d640 1 -- 172.21.15.83:0/99456225 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab980627b0 con 0x7fab980a98f0 2024-03-25T19:30:49.396 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.396+0000 7fab8f7fe640 1 -- 172.21.15.83:0/99456225 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fab88001cd0 con 0x7fab980a98f0 2024-03-25T19:30:49.396 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.396+0000 7fab8f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:49.396 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.396+0000 7fab8f7fe640 1 -- 172.21.15.83:0/99456225 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fab88004d90 con 0x7fab980a98f0 2024-03-25T19:30:49.396 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.396+0000 7fab8f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:49.396 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.396+0000 7fab8f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:49.396 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.396+0000 7fab8f7fe640 10 monclient: dump: 2024-03-25T19:30:49.396 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:30:49.397 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:30:49.397 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:49.397 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:49.397 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:49.397 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:30:49.397 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:30:49.397 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:30:49.397 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:30:49.397 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:49.397 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.396+0000 7fab95ffb640 4 set_mon_vals no callback set 2024-03-25T19:30:49.398 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.397+0000 7fab8f7fe640 1 -- 172.21.15.83:0/99456225 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7fab8801ad60 con 0x7fab980a98f0 2024-03-25T19:30:49.398 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.398+0000 7fab8ffff640 10 monclient: get_auth_request con 0x7fab7806dd60 auth_method 0 2024-03-25T19:30:49.398 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.398+0000 7fab8f7fe640 10 client.4722.objecter ms_handle_connect 0x7fab7806dd60 2024-03-25T19:30:49.403 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.403+0000 7fab8f7fe640 1 -- 172.21.15.83:0/99456225 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2659774754 0 0) 0x7fab880548a0 con 0x7fab980a98f0 2024-03-25T19:30:49.403 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.403+0000 7fab8f7fe640 10 client.4722.objecter ms_dispatch 0x7fab98000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:30:49.403 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.403+0000 7fab8f7fe640 3 client.4722.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:30:49.404 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.403+0000 7fab8f7fe640 3 client.4722.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:30:49.404 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.404+0000 7fab8f7fe640 20 client.4722.objecter dump_active .. 0 homeless 2024-03-25T19:30:49.404 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.404+0000 7fab8f7fe640 1 -- 172.21.15.83:0/99456225 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fab880503b0 con 0x7fab980a98f0 2024-03-25T19:30:49.404 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.404+0000 7fab8f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:49.404 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.404+0000 7fab8f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:49.623 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.624+0000 7fab9d60d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:49.624 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.624+0000 7fab9d60d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:49.624 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.624+0000 7fab9d60d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.83:6789/0 2024-03-25T19:30:49.624 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.624+0000 7fab9d60d640 1 -- 172.21.15.83:0/99456225 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fab981df360 con 0x7fab980a98f0 2024-03-25T19:30:49.668 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.668+0000 7fab8f7fe640 1 -- 172.21.15.83:0/99456225 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1070525753) 0x7fab880522e0 con 0x7fab980a98f0 2024-03-25T19:30:49.668 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.668+0000 7fab8f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:49.668 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.668+0000 7fab8f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:49.668 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:30:49.669 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":11,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:29:48.933916+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:48.933916+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:49.672 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.671+0000 7fab9d60d640 10 librados: watch_flush enter 2024-03-25T19:30:49.672 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.671+0000 7fab9d60d640 10 librados: watch_flush exit 2024-03-25T19:30:49.672 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.671+0000 7fab9d60d640 20 client.4722.objecter shutdown clearing up homeless session... 2024-03-25T19:30:49.672 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.671+0000 7fab9d60d640 10 client.4722.objecter successfully canceled tick 2024-03-25T19:30:49.672 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.672+0000 7fab9d60d640 1 -- 172.21.15.83:0/99456225 >> v1:172.21.15.103:6800/345072327 conn(0x7fab7806dd60 legacy=0x7fab78070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:49.672 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.672+0000 7fab9d60d640 10 monclient: shutdown 2024-03-25T19:30:49.672 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.672+0000 7fab9d60d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:49.672 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.672+0000 7fab9d60d640 1 -- 172.21.15.83:0/99456225 >> v1:172.21.15.83:6789/0 conn(0x7fab980a98f0 legacy=0x7fab981df6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:49.673 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.672+0000 7fab9d60d640 1 -- 172.21.15.83:0/99456225 shutdown_connections 2024-03-25T19:30:49.673 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.672+0000 7fab9d60d640 1 -- 172.21.15.83:0/99456225 >> 172.21.15.83:0/99456225 conn(0x7fab980b7ea0 msgr2=0x7fab98074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:49.673 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.672+0000 7fab9d60d640 1 -- 172.21.15.83:0/99456225 shutdown_connections 2024-03-25T19:30:49.673 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.672+0000 7fab9d60d640 1 -- 172.21.15.83:0/99456225 wait complete. 2024-03-25T19:30:49.673 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:49.672+0000 7fab9d60d640 1 librados: shutdown 2024-03-25T19:30:49.682 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:50.879 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:11:32.806 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:50.884 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:11:32.812 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:50.895 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:11:32.823 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:50.896 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:11:32.806 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:50.899 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:11:32.827 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:50.900 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:11:32.810 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:50.910 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:11:32.837 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:50.922 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:11:32.850 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:50.972 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:11:32.882 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:54.492 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:11:36.419 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:54.493 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:11:36.421 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:54.522 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:11:36.449 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:54.557 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:11:36.467 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:54.569 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:11:36.496 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:54.569 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:11:36.479 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:54.578 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:11:36.505 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:54.601 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:11:36.511 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:54.617 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:11:36.545 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:56.488 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:56.608 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.609+0000 7fee9d093640 10 monclient: get_monmap_and_config 2024-03-25T19:30:56.609 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.609+0000 7fee9d093640 10 monclient: build_initial_monmap 2024-03-25T19:30:56.609 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.609+0000 7fee9d093640 10 monclient: monmap: 2024-03-25T19:30:56.609 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:30:56.609 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:30:56.609 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:30:56.609982+0000 2024-03-25T19:30:56.609 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:30:56.609982+0000 2024-03-25T19:30:56.609 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:56.609 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:30:56.609 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:30:56.609 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:30:56.609 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:30:56.609 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:56.610 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.609+0000 7fee9d093640 1 Processor -- start 2024-03-25T19:30:56.610 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.610+0000 7fee9d093640 1 -- start start 2024-03-25T19:30:56.611 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.610+0000 7fee9d093640 10 monclient: init 2024-03-25T19:30:56.611 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.610+0000 7fee9d093640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:56.611 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.610+0000 7fee9d093640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:30:56.611 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.610+0000 7fee9d093640 10 monclient(hunting): picked mon.noname-c con 0x7fee98061890 addr v1:172.21.15.103:6789/0 2024-03-25T19:30:56.611 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.610+0000 7fee9d093640 10 monclient(hunting): picked mon.noname-a con 0x7fee9806e740 addr v1:172.21.15.83:6789/0 2024-03-25T19:30:56.611 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.610+0000 7fee9d093640 10 monclient(hunting): picked mon.noname-b con 0x7fee980ae280 addr v1:172.21.15.83:6790/0 2024-03-25T19:30:56.611 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.610+0000 7fee9d093640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee980ae830 con 0x7fee9806e740 2024-03-25T19:30:56.611 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.610+0000 7fee9d093640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee9806f720 con 0x7fee98061890 2024-03-25T19:30:56.611 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.610+0000 7fee9d093640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee980ada80 con 0x7fee980ae280 2024-03-25T19:30:56.611 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.610+0000 7fee9d093640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:56.611 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.610+0000 7fee9d093640 10 monclient(hunting): authenticate will time out at 6713.882697s 2024-03-25T19:30:56.617 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.617+0000 7fee96d9d640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fee98061890 0x7fee9813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:44526/0 (socket says 172.21.15.83:44526) 2024-03-25T19:30:56.617 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.617+0000 7fee96d9d640 1 -- 172.21.15.83:0/840114561 learned_addr learned my addr 172.21.15.83:0/840114561 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:30:56.631 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.632+0000 7fee95d9b640 1 -- 172.21.15.83:0/840114561 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1532620947 0 0) 0x7fee9806f720 con 0x7fee98061890 2024-03-25T19:30:56.632 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.632+0000 7fee95d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:30:56.632 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.632+0000 7fee95d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:56.632 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.632+0000 7fee95d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:56.632 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.632+0000 7fee95d9b640 10 monclient(hunting): my global_id is 4741 2024-03-25T19:30:56.632 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.632+0000 7fee95d9b640 1 -- 172.21.15.83:0/840114561 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee80003520 con 0x7fee98061890 2024-03-25T19:30:56.635 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.635+0000 7fee95d9b640 1 -- 172.21.15.83:0/840114561 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3425372543 0 0) 0x7fee980ae830 con 0x7fee9806e740 2024-03-25T19:30:56.635 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.635+0000 7fee95d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:30:56.635 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.635+0000 7fee95d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:56.635 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.635+0000 7fee95d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:56.635 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.635+0000 7fee95d9b640 10 monclient(hunting): my global_id is 4725 2024-03-25T19:30:56.635 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.635+0000 7fee95d9b640 1 -- 172.21.15.83:0/840114561 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee9806f720 con 0x7fee9806e740 2024-03-25T19:30:56.638 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.638+0000 7fee95d9b640 1 -- 172.21.15.83:0/840114561 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1523536656 0 0) 0x7fee980ada80 con 0x7fee980ae280 2024-03-25T19:30:56.638 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.638+0000 7fee95d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:30:56.638 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.638+0000 7fee95d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:56.638 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.638+0000 7fee95d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:56.638 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.638+0000 7fee95d9b640 10 monclient(hunting): my global_id is 4739 2024-03-25T19:30:56.638 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.638+0000 7fee95d9b640 1 -- 172.21.15.83:0/840114561 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee980ae830 con 0x7fee980ae280 2024-03-25T19:30:56.641 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.641+0000 7fee95d9b640 1 -- 172.21.15.83:0/840114561 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 250515168 0 0) 0x7fee80003520 con 0x7fee98061890 2024-03-25T19:30:56.642 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.641+0000 7fee95d9b640 1 -- 172.21.15.83:0/840114561 >> v1:172.21.15.83:6790/0 conn(0x7fee980ae280 legacy=0x7fee98145f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:56.642 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.641+0000 7fee95d9b640 1 -- 172.21.15.83:0/840114561 >> v1:172.21.15.83:6789/0 conn(0x7fee9806e740 legacy=0x7fee98145820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:56.642 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.641+0000 7fee95d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:30:56.642 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.641+0000 7fee95d9b640 1 monclient: found mon.noname-c 2024-03-25T19:30:56.642 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.641+0000 7fee95d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:56.642 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.641+0000 7fee95d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.103:6789/0 2024-03-25T19:30:56.642 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.641+0000 7fee95d9b640 1 -- 172.21.15.83:0/840114561 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee9813fbd0 con 0x7fee98061890 2024-03-25T19:30:56.642 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.641+0000 7fee95d9b640 10 monclient: _finish_auth 0 2024-03-25T19:30:56.642 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.641+0000 7fee95d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:30:56.642 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.641+0000 7fee95d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:56.642 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.641+0000 7fee95d9b640 1 -- 172.21.15.83:0/840114561 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fee88001690 con 0x7fee98061890 2024-03-25T19:30:56.642 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.641+0000 7fee9d093640 5 monclient: authenticate success, global_id 4741 2024-03-25T19:30:56.642 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.641+0000 7fee9d093640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:56.642 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.641+0000 7fee95d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:56.643 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.641+0000 7fee95d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:30:56.643 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.641+0000 7fee95d9b640 10 monclient: dump: 2024-03-25T19:30:56.643 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:30:56.643 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:30:56.643 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:56.643 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:56.643 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:56.643 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:30:56.643 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:30:56.643 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:30:56.643 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:30:56.643 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:56.643 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.642+0000 7fee9d093640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:56.649 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.649+0000 7fee95d9b640 1 -- 172.21.15.83:0/840114561 <== mon.1 v1:172.21.15.103:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee880017f0 con 0x7fee98061890 2024-03-25T19:30:56.649 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.649+0000 7fee95d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:56.649 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.649+0000 7fee95d9b640 1 -- 172.21.15.83:0/840114561 <== mon.1 v1:172.21.15.103:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fee88001c00 con 0x7fee98061890 2024-03-25T19:30:56.649 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.649+0000 7fee95d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:56.649 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.649+0000 7fee95d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:56.649 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.649+0000 7fee95d9b640 10 monclient: dump: 2024-03-25T19:30:56.649 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:30:56.649 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:30:56.649 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:56.649 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:56.650 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:56.650 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:30:56.650 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:30:56.650 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:30:56.650 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:30:56.650 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:56.650 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.649+0000 7fee9d093640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:56.650 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.649+0000 7fee9d093640 4 set_mon_vals no callback set 2024-03-25T19:30:56.652 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.650+0000 7fee9d093640 10 monclient: shutdown 2024-03-25T19:30:56.652 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.650+0000 7fee9d093640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:56.652 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.650+0000 7fee9d093640 1 -- 172.21.15.83:0/840114561 >> v1:172.21.15.103:6789/0 conn(0x7fee98061890 legacy=0x7fee9813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:56.652 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.650+0000 7fee9d093640 1 -- 172.21.15.83:0/840114561 shutdown_connections 2024-03-25T19:30:56.652 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.650+0000 7fee9d093640 1 -- 172.21.15.83:0/840114561 >> 172.21.15.83:0/840114561 conn(0x7fee980b0ef0 msgr2=0x7fee980b1300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:56.652 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.650+0000 7fee9d093640 1 -- 172.21.15.83:0/840114561 shutdown_connections 2024-03-25T19:30:56.652 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.650+0000 7fee9d093640 1 -- 172.21.15.83:0/840114561 wait complete. 2024-03-25T19:30:56.652 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.651+0000 7fee9d093640 10 monclient: build_initial_monmap 2024-03-25T19:30:56.652 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.651+0000 7fee9d093640 10 monclient: monmap: 2024-03-25T19:30:56.653 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:30:56.653 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:56.653 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:30:56.653 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:30:56.653 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:56.653 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:30:56.653 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:30:56.653 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:30:56.653 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:30:56.653 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:56.653 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.652+0000 7fee9d093640 1 librados: starting msgr at 2024-03-25T19:30:56.653 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.652+0000 7fee9d093640 1 librados: starting objecter 2024-03-25T19:30:56.653 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.652+0000 7fee9d093640 1 Processor -- start 2024-03-25T19:30:56.653 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.652+0000 7fee9d093640 1 -- start start 2024-03-25T19:30:56.653 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.652+0000 7fee9d093640 1 librados: setting wanted keys 2024-03-25T19:30:56.653 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.652+0000 7fee9d093640 1 librados: calling monclient init 2024-03-25T19:30:56.654 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.652+0000 7fee9d093640 10 monclient: init 2024-03-25T19:30:56.655 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.653+0000 7fee9d093640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:56.655 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.653+0000 7fee9d093640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:30:56.655 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.653+0000 7fee9d093640 10 monclient(hunting): picked mon.noname-b con 0x7fee98061890 addr v1:172.21.15.103:6789/0 2024-03-25T19:30:56.655 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.653+0000 7fee9d093640 10 monclient(hunting): picked mon.noname-a con 0x7fee980ae280 addr v1:172.21.15.83:6789/0 2024-03-25T19:30:56.655 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.653+0000 7fee9d093640 10 monclient(hunting): picked mon.noname-c con 0x7fee981d8190 addr v1:172.21.15.83:6790/0 2024-03-25T19:30:56.655 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.653+0000 7fee9d093640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee981d75a0 con 0x7fee980ae280 2024-03-25T19:30:56.655 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.653+0000 7fee9d093640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee98134dd0 con 0x7fee98061890 2024-03-25T19:30:56.655 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.653+0000 7fee9d093640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee981d3f70 con 0x7fee981d8190 2024-03-25T19:30:56.655 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.653+0000 7fee9d093640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:56.655 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.653+0000 7fee9d093640 10 monclient(hunting): authenticate will time out at 6713.925585s 2024-03-25T19:30:56.661 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.662+0000 7fee96d9d640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fee98061890 0x7fee981d7a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:44558/0 (socket says 172.21.15.83:44558) 2024-03-25T19:30:56.661 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.662+0000 7fee96d9d640 1 -- 172.21.15.83:0/72871116 learned_addr learned my addr 172.21.15.83:0/72871116 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:30:56.663 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.663+0000 7fee7f7fe640 10 client.?.objecter ms_handle_connect 0x7fee980ae280 2024-03-25T19:30:56.663 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.663+0000 7fee7f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:56.663 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.664+0000 7fee7f7fe640 10 client.?.objecter ms_handle_connect 0x7fee98061890 2024-03-25T19:30:56.664 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.664+0000 7fee7f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:56.670 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.670+0000 7fee7f7fe640 1 -- 172.21.15.83:0/72871116 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2495820323 0 0) 0x7fee981d75a0 con 0x7fee980ae280 2024-03-25T19:30:56.670 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.670+0000 7fee7f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:30:56.670 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.670+0000 7fee7f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:56.670 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.670+0000 7fee7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:56.670 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.670+0000 7fee7f7fe640 10 monclient(hunting): my global_id is 4728 2024-03-25T19:30:56.671 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.670+0000 7fee7f7fe640 1 -- 172.21.15.83:0/72871116 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee84004eb0 con 0x7fee980ae280 2024-03-25T19:30:56.671 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.671+0000 7fee7f7fe640 10 client.?.objecter ms_handle_connect 0x7fee981d8190 2024-03-25T19:30:56.671 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.671+0000 7fee7f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:56.675 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.675+0000 7fee7f7fe640 1 -- 172.21.15.83:0/72871116 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3912252700 0 0) 0x7fee98134dd0 con 0x7fee98061890 2024-03-25T19:30:56.675 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.675+0000 7fee7f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:30:56.675 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.675+0000 7fee7f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:56.675 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.675+0000 7fee7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:56.675 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.675+0000 7fee7f7fe640 10 monclient(hunting): my global_id is 4744 2024-03-25T19:30:56.675 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.675+0000 7fee7f7fe640 1 -- 172.21.15.83:0/72871116 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee981d75a0 con 0x7fee98061890 2024-03-25T19:30:56.682 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.683+0000 7fee7f7fe640 1 -- 172.21.15.83:0/72871116 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2284493675 0 0) 0x7fee981d3f70 con 0x7fee981d8190 2024-03-25T19:30:56.682 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.683+0000 7fee7f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:30:56.683 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.683+0000 7fee7f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:56.683 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.683+0000 7fee7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:56.683 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.683+0000 7fee7f7fe640 10 monclient(hunting): my global_id is 4742 2024-03-25T19:30:56.683 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.683+0000 7fee7f7fe640 1 -- 172.21.15.83:0/72871116 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee98134dd0 con 0x7fee981d8190 2024-03-25T19:30:56.696 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.696+0000 7fee7f7fe640 1 -- 172.21.15.83:0/72871116 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2809663753 0 0) 0x7fee981d75a0 con 0x7fee98061890 2024-03-25T19:30:56.696 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.697+0000 7fee7f7fe640 1 -- 172.21.15.83:0/72871116 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee981d3f70 con 0x7fee98061890 2024-03-25T19:30:56.696 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.697+0000 7fee7f7fe640 1 -- 172.21.15.83:0/72871116 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fee88001b00 con 0x7fee98061890 2024-03-25T19:30:56.697 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.697+0000 7fee7f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:56.697 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.697+0000 7fee7f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:30:56.697 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.697+0000 7fee7f7fe640 10 monclient(hunting): dump: 2024-03-25T19:30:56.697 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:30:56.697 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:30:56.697 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:56.697 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:56.697 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:56.697 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:30:56.697 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:30:56.697 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:30:56.697 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:30:56.697 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:56.697 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.697+0000 7fee7f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:56.700 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.700+0000 7fee7f7fe640 1 -- 172.21.15.83:0/72871116 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 50686525 0 0) 0x7fee84004eb0 con 0x7fee980ae280 2024-03-25T19:30:56.700 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.701+0000 7fee7f7fe640 1 -- 172.21.15.83:0/72871116 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee981d75a0 con 0x7fee980ae280 2024-03-25T19:30:56.700 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.701+0000 7fee7f7fe640 1 -- 172.21.15.83:0/72871116 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fee8c001970 con 0x7fee980ae280 2024-03-25T19:30:56.700 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.701+0000 7fee7f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:56.700 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.701+0000 7fee7f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:56.701 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.701+0000 7fee7f7fe640 10 monclient(hunting): dump: 2024-03-25T19:30:56.701 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:30:56.701 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:30:56.701 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:56.701 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:56.701 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:56.701 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:30:56.701 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:30:56.701 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:30:56.701 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:30:56.701 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:56.706 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.706+0000 7fee7f7fe640 1 -- 172.21.15.83:0/72871116 <== mon.1 v1:172.21.15.103:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1181068983 0 0) 0x7fee981d3f70 con 0x7fee98061890 2024-03-25T19:30:56.706 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.706+0000 7fee7f7fe640 1 -- 172.21.15.83:0/72871116 >> v1:172.21.15.83:6790/0 conn(0x7fee981d8190 legacy=0x7fee98135820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:56.706 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.706+0000 7fee7f7fe640 1 -- 172.21.15.83:0/72871116 >> v1:172.21.15.83:6789/0 conn(0x7fee980ae280 legacy=0x7fee981d4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:56.706 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.706+0000 7fee7f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:30:56.706 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.706+0000 7fee7f7fe640 1 monclient: found mon.b 2024-03-25T19:30:56.706 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.706+0000 7fee7f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:56.706 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.706+0000 7fee7f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:30:56.706 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.706+0000 7fee7f7fe640 1 -- 172.21.15.83:0/72871116 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee981d4100 con 0x7fee98061890 2024-03-25T19:30:56.706 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.706+0000 7fee7f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:30:56.706 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.706+0000 7fee7f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:30:56.706 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.706+0000 7fee7f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:56.707 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.706+0000 7fee9d093640 5 monclient: authenticate success, global_id 4744 2024-03-25T19:30:56.707 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.706+0000 7fee9d093640 10 monclient: _renew_subs 2024-03-25T19:30:56.707 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.706+0000 7fee9d093640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:30:56.707 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.706+0000 7fee9d093640 1 -- 172.21.15.83:0/72871116 --> v1:172.21.15.103:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee981d4c50 con 0x7fee98061890 2024-03-25T19:30:56.707 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.706+0000 7fee9d093640 10 client.4744.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:56.707 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.706+0000 7fee9d093640 10 monclient: _renew_subs 2024-03-25T19:30:56.707 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.706+0000 7fee9d093640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:30:56.707 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.706+0000 7fee9d093640 1 -- 172.21.15.83:0/72871116 --> v1:172.21.15.103:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fee98135f30 con 0x7fee98061890 2024-03-25T19:30:56.707 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.706+0000 7fee9d093640 1 librados: init done 2024-03-25T19:30:56.708 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.707+0000 7fee9d093640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:56.708 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.707+0000 7fee9d093640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:56.708 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.707+0000 7fee9d093640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:30:56.708 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.707+0000 7fee9d093640 1 -- 172.21.15.83:0/72871116 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee980627b0 con 0x7fee98061890 2024-03-25T19:30:56.996 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.996+0000 7fee7f7fe640 1 -- 172.21.15.83:0/72871116 <== mon.1 v1:172.21.15.103:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee88004a90 con 0x7fee98061890 2024-03-25T19:30:56.997 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.996+0000 7fee7f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:56.997 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.996+0000 7fee7f7fe640 1 -- 172.21.15.83:0/72871116 <== mon.1 v1:172.21.15.103:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fee88004da0 con 0x7fee98061890 2024-03-25T19:30:56.997 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.996+0000 7fee7f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:56.997 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.996+0000 7fee7f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:56.997 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.996+0000 7fee7f7fe640 10 monclient: dump: 2024-03-25T19:30:56.997 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:30:56.997 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:30:56.997 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:56.997 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:30:56.997 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:56.997 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:30:56.998 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:30:56.998 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:30:56.998 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:30:56.998 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:30:56.998 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.996+0000 7fee9559a640 4 set_mon_vals no callback set 2024-03-25T19:30:56.998 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.997+0000 7fee7f7fe640 1 -- 172.21.15.83:0/72871116 <== mon.1 v1:172.21.15.103:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7fee8801ad70 con 0x7fee98061890 2024-03-25T19:30:56.999 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.998+0000 7fee9659c640 10 monclient: get_auth_request con 0x7fee8406df70 auth_method 0 2024-03-25T19:30:56.999 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:56.998+0000 7fee7f7fe640 10 client.4744.objecter ms_handle_connect 0x7fee8406df70 2024-03-25T19:30:57.003 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:57.002+0000 7fee7f7fe640 1 -- 172.21.15.83:0/72871116 <== mon.1 v1:172.21.15.103:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2659774754 0 0) 0x7fee880548b0 con 0x7fee98061890 2024-03-25T19:30:57.003 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:57.002+0000 7fee7f7fe640 10 client.4744.objecter ms_dispatch 0x7fee98000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:30:57.003 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:57.002+0000 7fee7f7fe640 3 client.4744.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:30:57.003 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:57.002+0000 7fee7f7fe640 3 client.4744.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:30:57.003 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:57.003+0000 7fee7f7fe640 20 client.4744.objecter dump_active .. 0 homeless 2024-03-25T19:30:57.003 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:57.003+0000 7fee7f7fe640 1 -- 172.21.15.83:0/72871116 <== mon.1 v1:172.21.15.103:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fee880503c0 con 0x7fee98061890 2024-03-25T19:30:57.003 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:57.003+0000 7fee7f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:57.003 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:57.003+0000 7fee7f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:57.242 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:57.242+0000 7fee9d093640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:57.242 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:57.242+0000 7fee9d093640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:57.242 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:57.242+0000 7fee9d093640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:30:57.242 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:57.242+0000 7fee9d093640 1 -- 172.21.15.83:0/72871116 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee981d77a0 con 0x7fee98061890 2024-03-25T19:30:57.307 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:57.307+0000 7fee7f7fe640 1 -- 172.21.15.83:0/72871116 <== mon.1 v1:172.21.15.103:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1070525753) 0x7fee880522f0 con 0x7fee98061890 2024-03-25T19:30:57.307 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:57.307+0000 7fee7f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:57.307 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:57.307+0000 7fee7f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:57.307 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:30:57.308 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":11,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:29:48.933916+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:48.933916+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:57.309 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:57.309+0000 7fee9d093640 10 librados: watch_flush enter 2024-03-25T19:30:57.310 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:57.309+0000 7fee9d093640 10 librados: watch_flush exit 2024-03-25T19:30:57.310 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:57.309+0000 7fee9d093640 20 client.4744.objecter shutdown clearing up homeless session... 2024-03-25T19:30:57.310 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:57.309+0000 7fee9d093640 10 client.4744.objecter successfully canceled tick 2024-03-25T19:30:57.310 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:57.309+0000 7fee9d093640 1 -- 172.21.15.83:0/72871116 >> v1:172.21.15.103:6800/345072327 conn(0x7fee8406df70 legacy=0x7fee84070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:57.310 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:57.309+0000 7fee9d093640 10 monclient: shutdown 2024-03-25T19:30:57.310 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:57.309+0000 7fee9d093640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:57.310 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:57.309+0000 7fee9d093640 1 -- 172.21.15.83:0/72871116 >> v1:172.21.15.103:6789/0 conn(0x7fee98061890 legacy=0x7fee981d7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:57.310 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:57.309+0000 7fee9d093640 1 -- 172.21.15.83:0/72871116 shutdown_connections 2024-03-25T19:30:57.310 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:57.309+0000 7fee9d093640 1 -- 172.21.15.83:0/72871116 >> 172.21.15.83:0/72871116 conn(0x7fee980b0ef0 msgr2=0x7fee9813f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:57.310 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:57.309+0000 7fee9d093640 1 -- 172.21.15.83:0/72871116 shutdown_connections 2024-03-25T19:30:57.310 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:57.309+0000 7fee9d093640 1 -- 172.21.15.83:0/72871116 wait complete. 2024-03-25T19:30:57.310 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:30:57.310+0000 7fee9d093640 1 librados: shutdown 2024-03-25T19:30:57.321 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:31:04.128 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:31:04.235 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.235+0000 7fb3629af640 10 monclient: get_monmap_and_config 2024-03-25T19:31:04.235 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.235+0000 7fb3629af640 10 monclient: build_initial_monmap 2024-03-25T19:31:04.235 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.235+0000 7fb3629af640 10 monclient: monmap: 2024-03-25T19:31:04.235 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:31:04.235 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:31:04.235 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:31:04.236692+0000 2024-03-25T19:31:04.235 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:31:04.236692+0000 2024-03-25T19:31:04.235 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:04.236 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:31:04.236 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:31:04.236 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:31:04.236 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:31:04.236 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:31:04.237 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.236+0000 7fb3629af640 1 Processor -- start 2024-03-25T19:31:04.237 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.236+0000 7fb3629af640 1 -- start start 2024-03-25T19:31:04.237 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.236+0000 7fb3629af640 10 monclient: init 2024-03-25T19:31:04.237 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.237+0000 7fb3629af640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:04.237 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.237+0000 7fb3629af640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:31:04.237 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.237+0000 7fb3629af640 10 monclient(hunting): picked mon.noname-b con 0x7fb35c0a98f0 addr v1:172.21.15.83:6790/0 2024-03-25T19:31:04.237 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.237+0000 7fb3629af640 10 monclient(hunting): picked mon.noname-a con 0x7fb35c13f710 addr v1:172.21.15.83:6789/0 2024-03-25T19:31:04.237 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.237+0000 7fb3629af640 10 monclient(hunting): picked mon.noname-c con 0x7fb35c1430c0 addr v1:172.21.15.103:6789/0 2024-03-25T19:31:04.237 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.237+0000 7fb3629af640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb35c142520 con 0x7fb35c13f710 2024-03-25T19:31:04.237 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.237+0000 7fb3629af640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb35c0b28a0 con 0x7fb35c1430c0 2024-03-25T19:31:04.238 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.237+0000 7fb3629af640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb35c0b2a30 con 0x7fb35c0a98f0 2024-03-25T19:31:04.238 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.237+0000 7fb3629af640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:04.238 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.237+0000 7fb3629af640 10 monclient(hunting): authenticate will time out at 6721.509401s 2024-03-25T19:31:04.246 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.247+0000 7fb35b7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fb35c13f710 0x7fb35c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:54190/0 (socket says 172.21.15.83:54190) 2024-03-25T19:31:04.247 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.247+0000 7fb35b7fe640 1 -- 172.21.15.83:0/965105043 learned_addr learned my addr 172.21.15.83:0/965105043 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:31:04.262 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.262+0000 7fb35affd640 1 -- 172.21.15.83:0/965105043 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 284986093 0 0) 0x7fb35c142520 con 0x7fb35c13f710 2024-03-25T19:31:04.262 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.263+0000 7fb35affd640 10 monclient(hunting): _negotiate 2024-03-25T19:31:04.262 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.263+0000 7fb35affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:04.262 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.263+0000 7fb35affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:04.263 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.263+0000 7fb35affd640 10 monclient(hunting): my global_id is 4731 2024-03-25T19:31:04.263 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.263+0000 7fb35affd640 1 -- 172.21.15.83:0/965105043 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb344003520 con 0x7fb35c13f710 2024-03-25T19:31:04.269 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.269+0000 7fb35affd640 1 -- 172.21.15.83:0/965105043 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 196219876 0 0) 0x7fb35c0b28a0 con 0x7fb35c1430c0 2024-03-25T19:31:04.269 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.269+0000 7fb35affd640 10 monclient(hunting): _negotiate 2024-03-25T19:31:04.269 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.269+0000 7fb35affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:04.269 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.269+0000 7fb35affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:04.269 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.269+0000 7fb35affd640 10 monclient(hunting): my global_id is 4747 2024-03-25T19:31:04.269 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.269+0000 7fb35affd640 1 -- 172.21.15.83:0/965105043 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb35c142520 con 0x7fb35c1430c0 2024-03-25T19:31:04.289 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.288+0000 7fb35affd640 1 -- 172.21.15.83:0/965105043 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1739853765 0 0) 0x7fb35c142520 con 0x7fb35c1430c0 2024-03-25T19:31:04.289 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.288+0000 7fb35affd640 1 -- 172.21.15.83:0/965105043 >> v1:172.21.15.83:6790/0 conn(0x7fb35c0a98f0 legacy=0x7fb35c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:04.289 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.288+0000 7fb35affd640 1 -- 172.21.15.83:0/965105043 >> v1:172.21.15.83:6789/0 conn(0x7fb35c13f710 legacy=0x7fb35c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:04.289 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.288+0000 7fb35affd640 10 monclient: _finish_hunting 0 2024-03-25T19:31:04.289 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.288+0000 7fb35affd640 1 monclient: found mon.noname-c 2024-03-25T19:31:04.289 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.288+0000 7fb35affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:04.289 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.288+0000 7fb35affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.103:6789/0 2024-03-25T19:31:04.289 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.288+0000 7fb35affd640 1 -- 172.21.15.83:0/965105043 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb35c0b44c0 con 0x7fb35c1430c0 2024-03-25T19:31:04.289 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.288+0000 7fb35affd640 10 monclient: _finish_auth 0 2024-03-25T19:31:04.289 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.288+0000 7fb35affd640 10 monclient: _check_auth_tickets 2024-03-25T19:31:04.289 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.288+0000 7fb35affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:04.289 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.288+0000 7fb35affd640 1 -- 172.21.15.83:0/965105043 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fb3540019f0 con 0x7fb35c1430c0 2024-03-25T19:31:04.289 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.288+0000 7fb3629af640 5 monclient: authenticate success, global_id 4747 2024-03-25T19:31:04.289 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.288+0000 7fb3629af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:04.290 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.288+0000 7fb35affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:04.290 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.288+0000 7fb35affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:31:04.290 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.288+0000 7fb35affd640 10 monclient: dump: 2024-03-25T19:31:04.290 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:31:04.290 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:31:04.290 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:31:04.290 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:31:04.290 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:04.290 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:31:04.290 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:31:04.290 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:31:04.290 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:31:04.290 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:31:04.290 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.288+0000 7fb3629af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:04.305 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.305+0000 7fb35affd640 1 -- 172.21.15.83:0/965105043 <== mon.1 v1:172.21.15.103:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb354001b50 con 0x7fb35c1430c0 2024-03-25T19:31:04.305 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.305+0000 7fb35affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:04.305 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.305+0000 7fb35affd640 1 -- 172.21.15.83:0/965105043 <== mon.1 v1:172.21.15.103:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fb354001f20 con 0x7fb35c1430c0 2024-03-25T19:31:04.305 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.305+0000 7fb35affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:04.305 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.305+0000 7fb35affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:04.305 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.305+0000 7fb35affd640 10 monclient: dump: 2024-03-25T19:31:04.305 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:31:04.305 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:31:04.305 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:31:04.306 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:31:04.306 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:04.306 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:31:04.306 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:31:04.306 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:31:04.306 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:31:04.306 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:31:04.306 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.305+0000 7fb3629af640 10 monclient: get_monmap_and_config success 2024-03-25T19:31:04.306 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.305+0000 7fb3629af640 4 set_mon_vals no callback set 2024-03-25T19:31:04.308 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.306+0000 7fb3629af640 10 monclient: shutdown 2024-03-25T19:31:04.308 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.306+0000 7fb3629af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:04.308 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.306+0000 7fb3629af640 1 -- 172.21.15.83:0/965105043 >> v1:172.21.15.103:6789/0 conn(0x7fb35c1430c0 legacy=0x7fb35c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:04.308 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.306+0000 7fb3629af640 1 -- 172.21.15.83:0/965105043 shutdown_connections 2024-03-25T19:31:04.308 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.306+0000 7fb3629af640 1 -- 172.21.15.83:0/965105043 >> 172.21.15.83:0/965105043 conn(0x7fb35c0b7ea0 msgr2=0x7fb35c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:04.308 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.306+0000 7fb3629af640 1 -- 172.21.15.83:0/965105043 shutdown_connections 2024-03-25T19:31:04.308 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.306+0000 7fb3629af640 1 -- 172.21.15.83:0/965105043 wait complete. 2024-03-25T19:31:04.309 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.307+0000 7fb3629af640 10 monclient: build_initial_monmap 2024-03-25T19:31:04.309 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.307+0000 7fb3629af640 10 monclient: monmap: 2024-03-25T19:31:04.309 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:31:04.309 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:31:04.309 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:31:04.309 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:31:04.309 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:04.309 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:31:04.309 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:31:04.309 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:31:04.309 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:31:04.309 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:31:04.309 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.307+0000 7fb3629af640 1 librados: starting msgr at 2024-03-25T19:31:04.309 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.307+0000 7fb3629af640 1 librados: starting objecter 2024-03-25T19:31:04.309 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.307+0000 7fb3629af640 1 Processor -- start 2024-03-25T19:31:04.309 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.308+0000 7fb3629af640 1 -- start start 2024-03-25T19:31:04.309 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.308+0000 7fb3629af640 1 librados: setting wanted keys 2024-03-25T19:31:04.310 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.308+0000 7fb3629af640 1 librados: calling monclient init 2024-03-25T19:31:04.310 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.308+0000 7fb3629af640 10 monclient: init 2024-03-25T19:31:04.310 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.308+0000 7fb3629af640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:04.310 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.308+0000 7fb3629af640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:31:04.310 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.308+0000 7fb3629af640 10 monclient(hunting): picked mon.noname-c con 0x7fb35c0a98f0 addr v1:172.21.15.83:6790/0 2024-03-25T19:31:04.310 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.308+0000 7fb3629af640 10 monclient(hunting): picked mon.noname-a con 0x7fb35c13f710 addr v1:172.21.15.83:6789/0 2024-03-25T19:31:04.310 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.308+0000 7fb3629af640 10 monclient(hunting): picked mon.noname-b con 0x7fb35c1430c0 addr v1:172.21.15.103:6789/0 2024-03-25T19:31:04.310 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.308+0000 7fb3629af640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb35c0abae0 con 0x7fb35c13f710 2024-03-25T19:31:04.310 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.308+0000 7fb3629af640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb35c1d4380 con 0x7fb35c1430c0 2024-03-25T19:31:04.310 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.308+0000 7fb3629af640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb35c138f10 con 0x7fb35c0a98f0 2024-03-25T19:31:04.310 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.308+0000 7fb3629af640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:04.310 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.308+0000 7fb3629af640 10 monclient(hunting): authenticate will time out at 6721.580389s 2024-03-25T19:31:04.316 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.317+0000 7fb360f4c640 1 --1- >> v1:172.21.15.103:6789/0 conn(0x7fb35c1430c0 0x7fb35c133ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.103:6789/0 says I am v1:172.21.15.83:49380/0 (socket says 172.21.15.83:49380) 2024-03-25T19:31:04.316 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.317+0000 7fb360f4c640 1 -- 172.21.15.83:0/2507441645 learned_addr learned my addr 172.21.15.83:0/2507441645 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:31:04.319 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.319+0000 7fb358ff9640 10 client.?.objecter ms_handle_connect 0x7fb35c13f710 2024-03-25T19:31:04.319 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.319+0000 7fb358ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:04.322 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.322+0000 7fb358ff9640 10 client.?.objecter ms_handle_connect 0x7fb35c1430c0 2024-03-25T19:31:04.322 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.322+0000 7fb358ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:04.331 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.331+0000 7fb358ff9640 1 -- 172.21.15.83:0/2507441645 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 272251253 0 0) 0x7fb35c1d4380 con 0x7fb35c1430c0 2024-03-25T19:31:04.331 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.331+0000 7fb358ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:31:04.331 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.331+0000 7fb358ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:04.331 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.331+0000 7fb358ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:04.331 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.331+0000 7fb358ff9640 10 monclient(hunting): my global_id is 4750 2024-03-25T19:31:04.331 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.331+0000 7fb358ff9640 1 -- 172.21.15.83:0/2507441645 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb348004eb0 con 0x7fb35c1430c0 2024-03-25T19:31:04.332 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.333+0000 7fb358ff9640 10 client.?.objecter ms_handle_connect 0x7fb35c0a98f0 2024-03-25T19:31:04.332 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.333+0000 7fb358ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:04.333 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.334+0000 7fb358ff9640 1 -- 172.21.15.83:0/2507441645 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 873169454 0 0) 0x7fb35c0abae0 con 0x7fb35c13f710 2024-03-25T19:31:04.333 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.334+0000 7fb358ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:31:04.334 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.334+0000 7fb358ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:04.334 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.334+0000 7fb358ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:04.334 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.334+0000 7fb358ff9640 10 monclient(hunting): my global_id is 4734 2024-03-25T19:31:04.334 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.334+0000 7fb358ff9640 1 -- 172.21.15.83:0/2507441645 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb35c1d4380 con 0x7fb35c13f710 2024-03-25T19:31:04.341 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.342+0000 7fb358ff9640 1 -- 172.21.15.83:0/2507441645 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1448117421 0 0) 0x7fb35c138f10 con 0x7fb35c0a98f0 2024-03-25T19:31:04.341 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.342+0000 7fb358ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:31:04.341 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.342+0000 7fb358ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:04.342 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.342+0000 7fb358ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:04.342 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.342+0000 7fb358ff9640 10 monclient(hunting): my global_id is 4748 2024-03-25T19:31:04.342 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.342+0000 7fb358ff9640 1 -- 172.21.15.83:0/2507441645 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb35c0abae0 con 0x7fb35c0a98f0 2024-03-25T19:31:04.344 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.344+0000 7fb358ff9640 1 -- 172.21.15.83:0/2507441645 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 674042752 0 0) 0x7fb348004eb0 con 0x7fb35c1430c0 2024-03-25T19:31:04.344 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.344+0000 7fb358ff9640 1 -- 172.21.15.83:0/2507441645 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb35c138f10 con 0x7fb35c1430c0 2024-03-25T19:31:04.344 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.344+0000 7fb358ff9640 1 -- 172.21.15.83:0/2507441645 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fb354001f60 con 0x7fb35c1430c0 2024-03-25T19:31:04.344 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.344+0000 7fb358ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:04.344 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.344+0000 7fb358ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:31:04.344 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.344+0000 7fb358ff9640 10 monclient(hunting): dump: 2024-03-25T19:31:04.344 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:31:04.344 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:31:04.344 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:31:04.345 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:31:04.345 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:04.345 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:31:04.345 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:31:04.345 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:31:04.345 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:31:04.345 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:31:04.345 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.344+0000 7fb358ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:31:04.351 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.352+0000 7fb358ff9640 1 -- 172.21.15.83:0/2507441645 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4283378600 0 0) 0x7fb35c1d4380 con 0x7fb35c13f710 2024-03-25T19:31:04.351 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.352+0000 7fb358ff9640 1 -- 172.21.15.83:0/2507441645 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb348004eb0 con 0x7fb35c13f710 2024-03-25T19:31:04.351 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.352+0000 7fb358ff9640 1 -- 172.21.15.83:0/2507441645 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fb350001950 con 0x7fb35c13f710 2024-03-25T19:31:04.351 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.352+0000 7fb358ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:04.352 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.352+0000 7fb358ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:31:04.352 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.352+0000 7fb358ff9640 10 monclient(hunting): dump: 2024-03-25T19:31:04.352 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:31:04.352 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:31:04.352 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:31:04.352 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:31:04.352 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:04.352 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:31:04.352 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:31:04.352 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:31:04.352 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:31:04.352 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:31:04.353 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.353+0000 7fb358ff9640 1 -- 172.21.15.83:0/2507441645 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 802522247 0 0) 0x7fb35c0abae0 con 0x7fb35c0a98f0 2024-03-25T19:31:04.353 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.353+0000 7fb358ff9640 1 -- 172.21.15.83:0/2507441645 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb35c1d4380 con 0x7fb35c0a98f0 2024-03-25T19:31:04.353 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.353+0000 7fb358ff9640 1 -- 172.21.15.83:0/2507441645 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fb34c001870 con 0x7fb35c0a98f0 2024-03-25T19:31:04.353 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.353+0000 7fb358ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:04.353 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.353+0000 7fb358ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:31:04.353 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.353+0000 7fb358ff9640 10 monclient(hunting): dump: 2024-03-25T19:31:04.353 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:31:04.353 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:31:04.354 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:31:04.354 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:31:04.354 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:04.354 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:31:04.354 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:31:04.354 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:31:04.354 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:31:04.354 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:31:04.357 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.357+0000 7fb358ff9640 1 -- 172.21.15.83:0/2507441645 <== mon.1 v1:172.21.15.103:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2208022150 0 0) 0x7fb35c138f10 con 0x7fb35c1430c0 2024-03-25T19:31:04.357 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.357+0000 7fb358ff9640 1 -- 172.21.15.83:0/2507441645 >> v1:172.21.15.83:6790/0 conn(0x7fb35c0a98f0 legacy=0x7fb35c138800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:04.357 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.357+0000 7fb358ff9640 1 -- 172.21.15.83:0/2507441645 >> v1:172.21.15.83:6789/0 conn(0x7fb35c13f710 legacy=0x7fb35c1d48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:04.357 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.357+0000 7fb358ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:31:04.357 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.357+0000 7fb358ff9640 1 monclient: found mon.b 2024-03-25T19:31:04.357 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.357+0000 7fb358ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:04.357 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.357+0000 7fb358ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:31:04.358 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.357+0000 7fb358ff9640 1 -- 172.21.15.83:0/2507441645 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb35c1390a0 con 0x7fb35c1430c0 2024-03-25T19:31:04.358 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.357+0000 7fb358ff9640 10 monclient: _finish_auth 0 2024-03-25T19:31:04.358 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.357+0000 7fb358ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:31:04.358 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.357+0000 7fb358ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:04.358 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.357+0000 7fb3629af640 5 monclient: authenticate success, global_id 4750 2024-03-25T19:31:04.358 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.357+0000 7fb3629af640 10 monclient: _renew_subs 2024-03-25T19:31:04.358 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.357+0000 7fb3629af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:31:04.358 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.357+0000 7fb3629af640 1 -- 172.21.15.83:0/2507441645 --> v1:172.21.15.103:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb35c135040 con 0x7fb35c1430c0 2024-03-25T19:31:04.358 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.357+0000 7fb3629af640 10 client.4750.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:31:04.358 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.358+0000 7fb3629af640 10 monclient: _renew_subs 2024-03-25T19:31:04.358 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.358+0000 7fb3629af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:31:04.358 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.358+0000 7fb3629af640 1 -- 172.21.15.83:0/2507441645 --> v1:172.21.15.103:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb35c1d5190 con 0x7fb35c1430c0 2024-03-25T19:31:04.358 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.358+0000 7fb3629af640 1 librados: init done 2024-03-25T19:31:04.359 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.359+0000 7fb3629af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:04.359 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.359+0000 7fb3629af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:04.359 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.359+0000 7fb3629af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:31:04.359 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.359+0000 7fb3629af640 1 -- 172.21.15.83:0/2507441645 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb35c0627b0 con 0x7fb35c1430c0 2024-03-25T19:31:04.579 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.579+0000 7fb358ff9640 1 -- 172.21.15.83:0/2507441645 <== mon.1 v1:172.21.15.103:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb354003c20 con 0x7fb35c1430c0 2024-03-25T19:31:04.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.579+0000 7fb358ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:04.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.579+0000 7fb358ff9640 1 -- 172.21.15.83:0/2507441645 <== mon.1 v1:172.21.15.103:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fb354003fb0 con 0x7fb35c1430c0 2024-03-25T19:31:04.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.579+0000 7fb358ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:04.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.579+0000 7fb358ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:04.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.579+0000 7fb358ff9640 10 monclient: dump: 2024-03-25T19:31:04.580 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:31:04.580 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:31:04.580 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:31:04.580 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:31:04.580 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:04.580 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:31:04.580 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:31:04.580 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:31:04.581 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:31:04.581 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:31:04.581 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.579+0000 7fb35a7fc640 4 set_mon_vals no callback set 2024-03-25T19:31:04.638 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.638+0000 7fb358ff9640 1 -- 172.21.15.83:0/2507441645 <== mon.1 v1:172.21.15.103:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7fb354004110 con 0x7fb35c1430c0 2024-03-25T19:31:04.638 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.639+0000 7fb35bfff640 10 monclient: get_auth_request con 0x7fb34806e0f0 auth_method 0 2024-03-25T19:31:04.638 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.639+0000 7fb358ff9640 10 client.4750.objecter ms_handle_connect 0x7fb34806e0f0 2024-03-25T19:31:04.642 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.642+0000 7fb358ff9640 1 -- 172.21.15.83:0/2507441645 <== mon.1 v1:172.21.15.103:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2659774754 0 0) 0x7fb354054be0 con 0x7fb35c1430c0 2024-03-25T19:31:04.642 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.642+0000 7fb358ff9640 10 client.4750.objecter ms_dispatch 0x7fb35c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:31:04.642 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.642+0000 7fb358ff9640 3 client.4750.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:31:04.642 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.642+0000 7fb358ff9640 3 client.4750.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:31:04.642 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.642+0000 7fb358ff9640 20 client.4750.objecter dump_active .. 0 homeless 2024-03-25T19:31:04.644 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.645+0000 7fb358ff9640 1 -- 172.21.15.83:0/2507441645 <== mon.1 v1:172.21.15.103:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb3540506f0 con 0x7fb35c1430c0 2024-03-25T19:31:04.644 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.645+0000 7fb358ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:04.644 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.645+0000 7fb358ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:31:04.890 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.890+0000 7fb3629af640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:04.890 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.890+0000 7fb3629af640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:04.890 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.890+0000 7fb3629af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.103:6789/0 2024-03-25T19:31:04.890 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.890+0000 7fb3629af640 1 -- 172.21.15.83:0/2507441645 --> v1:172.21.15.103:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb35c1d41e0 con 0x7fb35c1430c0 2024-03-25T19:31:04.910 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.910+0000 7fb358ff9640 1 -- 172.21.15.83:0/2507441645 <== mon.1 v1:172.21.15.103:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1070525753) 0x7fb35c1d41e0 con 0x7fb35c1430c0 2024-03-25T19:31:04.910 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.910+0000 7fb358ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:04.911 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.910+0000 7fb358ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:31:04.911 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:31:04.911 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":11,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:29:48.933916+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:48.933916+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:31:04.913 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.912+0000 7fb3629af640 10 librados: watch_flush enter 2024-03-25T19:31:04.913 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.912+0000 7fb3629af640 10 librados: watch_flush exit 2024-03-25T19:31:04.913 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.912+0000 7fb3629af640 20 client.4750.objecter shutdown clearing up homeless session... 2024-03-25T19:31:04.913 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.912+0000 7fb3629af640 10 client.4750.objecter successfully canceled tick 2024-03-25T19:31:04.913 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.912+0000 7fb3629af640 1 -- 172.21.15.83:0/2507441645 >> v1:172.21.15.103:6800/345072327 conn(0x7fb34806e0f0 legacy=0x7fb3480705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:04.913 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.912+0000 7fb3629af640 10 monclient: shutdown 2024-03-25T19:31:04.913 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.912+0000 7fb3629af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:04.913 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.912+0000 7fb3629af640 1 -- 172.21.15.83:0/2507441645 >> v1:172.21.15.103:6789/0 conn(0x7fb35c1430c0 legacy=0x7fb35c133ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:04.913 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.912+0000 7fb3629af640 1 -- 172.21.15.83:0/2507441645 shutdown_connections 2024-03-25T19:31:04.914 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.912+0000 7fb3629af640 1 -- 172.21.15.83:0/2507441645 >> 172.21.15.83:0/2507441645 conn(0x7fb35c0b7ea0 msgr2=0x7fb35c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:04.914 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.912+0000 7fb3629af640 1 -- 172.21.15.83:0/2507441645 shutdown_connections 2024-03-25T19:31:04.914 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.912+0000 7fb3629af640 1 -- 172.21.15.83:0/2507441645 wait complete. 2024-03-25T19:31:04.914 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:04.912+0000 7fb3629af640 1 librados: shutdown 2024-03-25T19:31:04.925 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:31:11.732 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:31:11.848 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.848+0000 7fc3efb6d640 10 monclient: get_monmap_and_config 2024-03-25T19:31:11.848 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.848+0000 7fc3efb6d640 10 monclient: build_initial_monmap 2024-03-25T19:31:11.848 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.848+0000 7fc3efb6d640 10 monclient: monmap: 2024-03-25T19:31:11.848 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:31:11.848 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:31:11.848 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:31:11.849446+0000 2024-03-25T19:31:11.848 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:31:11.849446+0000 2024-03-25T19:31:11.848 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:11.849 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:31:11.849 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:31:11.849 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-c 2024-03-25T19:31:11.849 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-b 2024-03-25T19:31:11.849 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:31:11.850 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.849+0000 7fc3efb6d640 1 Processor -- start 2024-03-25T19:31:11.850 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.849+0000 7fc3efb6d640 1 -- start start 2024-03-25T19:31:11.851 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.849+0000 7fc3efb6d640 10 monclient: init 2024-03-25T19:31:11.851 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.850+0000 7fc3efb6d640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:11.851 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.850+0000 7fc3efb6d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:31:11.851 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.850+0000 7fc3efb6d640 10 monclient(hunting): picked mon.noname-c con 0x7fc3e80a98f0 addr v1:172.21.15.103:6789/0 2024-03-25T19:31:11.851 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.850+0000 7fc3efb6d640 10 monclient(hunting): picked mon.noname-a con 0x7fc3e813f710 addr v1:172.21.15.83:6789/0 2024-03-25T19:31:11.851 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.850+0000 7fc3efb6d640 10 monclient(hunting): picked mon.noname-b con 0x7fc3e81430c0 addr v1:172.21.15.83:6790/0 2024-03-25T19:31:11.851 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.850+0000 7fc3efb6d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e8142520 con 0x7fc3e813f710 2024-03-25T19:31:11.851 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.850+0000 7fc3efb6d640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e80b28a0 con 0x7fc3e80a98f0 2024-03-25T19:31:11.851 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.850+0000 7fc3efb6d640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e80b2a30 con 0x7fc3e81430c0 2024-03-25T19:31:11.851 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.850+0000 7fc3efb6d640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:11.851 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.850+0000 7fc3efb6d640 10 monclient(hunting): authenticate will time out at 6729.122435s 2024-03-25T19:31:11.861 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.861+0000 7fc3ee10a640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fc3e81430c0 0x7fc3e814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:45494/0 (socket says 172.21.15.83:45494) 2024-03-25T19:31:11.861 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.861+0000 7fc3ee10a640 1 -- 172.21.15.83:0/3471486530 learned_addr learned my addr 172.21.15.83:0/3471486530 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:31:11.880 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.880+0000 7fc3ec907640 1 -- 172.21.15.83:0/3471486530 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3073351012 0 0) 0x7fc3e80b28a0 con 0x7fc3e80a98f0 2024-03-25T19:31:11.880 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.880+0000 7fc3ec907640 10 monclient(hunting): _negotiate 2024-03-25T19:31:11.880 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.880+0000 7fc3ec907640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:11.880 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.880+0000 7fc3ec907640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:11.881 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.880+0000 7fc3ec907640 10 monclient(hunting): my global_id is 4753 2024-03-25T19:31:11.881 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.880+0000 7fc3ec907640 1 -- 172.21.15.83:0/3471486530 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3d4003520 con 0x7fc3e80a98f0 2024-03-25T19:31:11.881 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.880+0000 7fc3ec907640 1 -- 172.21.15.83:0/3471486530 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2490365576 0 0) 0x7fc3e80b2a30 con 0x7fc3e81430c0 2024-03-25T19:31:11.881 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.880+0000 7fc3ec907640 10 monclient(hunting): _negotiate 2024-03-25T19:31:11.881 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.880+0000 7fc3ec907640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:11.881 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.880+0000 7fc3ec907640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:11.881 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.880+0000 7fc3ec907640 10 monclient(hunting): my global_id is 4751 2024-03-25T19:31:11.881 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.880+0000 7fc3ec907640 1 -- 172.21.15.83:0/3471486530 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3e80b28a0 con 0x7fc3e81430c0 2024-03-25T19:31:11.883 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.883+0000 7fc3ec907640 1 -- 172.21.15.83:0/3471486530 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 100737442 0 0) 0x7fc3e8142520 con 0x7fc3e813f710 2024-03-25T19:31:11.883 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.883+0000 7fc3ec907640 10 monclient(hunting): _negotiate 2024-03-25T19:31:11.883 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.883+0000 7fc3ec907640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:11.883 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.883+0000 7fc3ec907640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:11.883 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.883+0000 7fc3ec907640 10 monclient(hunting): my global_id is 4737 2024-03-25T19:31:11.883 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.883+0000 7fc3ec907640 1 -- 172.21.15.83:0/3471486530 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3e80b2a30 con 0x7fc3e813f710 2024-03-25T19:31:11.888 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.888+0000 7fc3ec907640 1 -- 172.21.15.83:0/3471486530 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 159478168 0 0) 0x7fc3d4003520 con 0x7fc3e80a98f0 2024-03-25T19:31:11.888 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.888+0000 7fc3ec907640 1 -- 172.21.15.83:0/3471486530 >> v1:172.21.15.83:6790/0 conn(0x7fc3e81430c0 legacy=0x7fc3e814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:11.888 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.888+0000 7fc3ec907640 1 -- 172.21.15.83:0/3471486530 >> v1:172.21.15.83:6789/0 conn(0x7fc3e813f710 legacy=0x7fc3e81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:11.888 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.888+0000 7fc3ec907640 10 monclient: _finish_hunting 0 2024-03-25T19:31:11.888 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.888+0000 7fc3ec907640 1 monclient: found mon.noname-c 2024-03-25T19:31:11.889 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.888+0000 7fc3ec907640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:11.889 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.888+0000 7fc3ec907640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.103:6789/0 2024-03-25T19:31:11.889 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.888+0000 7fc3ec907640 1 -- 172.21.15.83:0/3471486530 --> v1:172.21.15.103:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e80b44c0 con 0x7fc3e80a98f0 2024-03-25T19:31:11.889 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.888+0000 7fc3ec907640 10 monclient: _finish_auth 0 2024-03-25T19:31:11.889 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.888+0000 7fc3ec907640 10 monclient: _check_auth_tickets 2024-03-25T19:31:11.889 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.888+0000 7fc3ec907640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:11.889 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.888+0000 7fc3ec907640 1 -- 172.21.15.83:0/3471486530 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fc3dc001690 con 0x7fc3e80a98f0 2024-03-25T19:31:11.889 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.888+0000 7fc3efb6d640 5 monclient: authenticate success, global_id 4753 2024-03-25T19:31:11.889 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.888+0000 7fc3efb6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:11.889 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.888+0000 7fc3ec907640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:11.889 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.888+0000 7fc3ec907640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:31:11.889 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.888+0000 7fc3ec907640 10 monclient: dump: 2024-03-25T19:31:11.889 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:31:11.890 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:31:11.890 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:31:11.890 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:31:11.890 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:11.890 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:31:11.890 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:31:11.890 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:31:11.890 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:31:11.890 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:31:11.890 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.888+0000 7fc3efb6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:11.904 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.904+0000 7fc3ec907640 1 -- 172.21.15.83:0/3471486530 <== mon.1 v1:172.21.15.103:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3dc0017f0 con 0x7fc3e80a98f0 2024-03-25T19:31:11.904 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.904+0000 7fc3ec907640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:11.904 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.904+0000 7fc3ec907640 1 -- 172.21.15.83:0/3471486530 <== mon.1 v1:172.21.15.103:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fc3dc001c00 con 0x7fc3e80a98f0 2024-03-25T19:31:11.904 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.904+0000 7fc3ec907640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:11.904 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.904+0000 7fc3ec907640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:11.904 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.904+0000 7fc3ec907640 10 monclient: dump: 2024-03-25T19:31:11.904 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:31:11.904 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:31:11.904 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:31:11.904 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:31:11.904 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:11.905 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:31:11.905 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:31:11.905 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:31:11.905 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:31:11.905 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:31:11.905 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.904+0000 7fc3efb6d640 10 monclient: get_monmap_and_config success 2024-03-25T19:31:11.905 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.904+0000 7fc3efb6d640 4 set_mon_vals no callback set 2024-03-25T19:31:11.907 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.905+0000 7fc3efb6d640 10 monclient: shutdown 2024-03-25T19:31:11.907 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.905+0000 7fc3efb6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:11.907 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.905+0000 7fc3efb6d640 1 -- 172.21.15.83:0/3471486530 >> v1:172.21.15.103:6789/0 conn(0x7fc3e80a98f0 legacy=0x7fc3e80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:11.907 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.905+0000 7fc3efb6d640 1 -- 172.21.15.83:0/3471486530 shutdown_connections 2024-03-25T19:31:11.907 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.905+0000 7fc3efb6d640 1 -- 172.21.15.83:0/3471486530 >> 172.21.15.83:0/3471486530 conn(0x7fc3e80b7ea0 msgr2=0x7fc3e80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:11.907 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.905+0000 7fc3efb6d640 1 -- 172.21.15.83:0/3471486530 shutdown_connections 2024-03-25T19:31:11.907 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.905+0000 7fc3efb6d640 1 -- 172.21.15.83:0/3471486530 wait complete. 2024-03-25T19:31:11.908 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.905+0000 7fc3efb6d640 10 monclient: build_initial_monmap 2024-03-25T19:31:11.908 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.906+0000 7fc3efb6d640 10 monclient: monmap: 2024-03-25T19:31:11.908 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T19:31:11.908 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:31:11.908 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T19:31:11.908 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T19:31:11.908 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:11.908 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T19:31:11.908 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a 2024-03-25T19:31:11.908 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.noname-b 2024-03-25T19:31:11.908 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.noname-c 2024-03-25T19:31:11.908 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:31:11.909 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.906+0000 7fc3efb6d640 1 librados: starting msgr at 2024-03-25T19:31:11.909 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.906+0000 7fc3efb6d640 1 librados: starting objecter 2024-03-25T19:31:11.909 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.906+0000 7fc3efb6d640 1 Processor -- start 2024-03-25T19:31:11.909 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.906+0000 7fc3efb6d640 1 -- start start 2024-03-25T19:31:11.909 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.906+0000 7fc3efb6d640 1 librados: setting wanted keys 2024-03-25T19:31:11.909 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.906+0000 7fc3efb6d640 1 librados: calling monclient init 2024-03-25T19:31:11.909 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.906+0000 7fc3efb6d640 10 monclient: init 2024-03-25T19:31:11.909 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.907+0000 7fc3efb6d640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:11.909 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.907+0000 7fc3efb6d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:31:11.909 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.907+0000 7fc3efb6d640 10 monclient(hunting): picked mon.noname-a con 0x7fc3e80a98f0 addr v1:172.21.15.83:6789/0 2024-03-25T19:31:11.909 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.907+0000 7fc3efb6d640 10 monclient(hunting): picked mon.noname-c con 0x7fc3e813f710 addr v1:172.21.15.83:6790/0 2024-03-25T19:31:11.909 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.907+0000 7fc3efb6d640 10 monclient(hunting): picked mon.noname-b con 0x7fc3e81430c0 addr v1:172.21.15.103:6789/0 2024-03-25T19:31:11.909 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.907+0000 7fc3efb6d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e80704d0 con 0x7fc3e80a98f0 2024-03-25T19:31:11.909 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.907+0000 7fc3efb6d640 1 -- --> v1:172.21.15.103:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e81df0d0 con 0x7fc3e81430c0 2024-03-25T19:31:11.909 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.907+0000 7fc3efb6d640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e806f490 con 0x7fc3e813f710 2024-03-25T19:31:11.910 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.907+0000 7fc3efb6d640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:11.910 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.907+0000 7fc3efb6d640 10 monclient(hunting): authenticate will time out at 6729.179126s 2024-03-25T19:31:11.913 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.914+0000 7fc3ed108640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fc3e813f710 0x7fc3e81df5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:45512/0 (socket says 172.21.15.83:45512) 2024-03-25T19:31:11.913 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.914+0000 7fc3ed108640 1 -- 172.21.15.83:0/3569736411 learned_addr learned my addr 172.21.15.83:0/3569736411 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T19:31:11.918 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.919+0000 7fc3da7fc640 10 client.?.objecter ms_handle_connect 0x7fc3e80a98f0 2024-03-25T19:31:11.918 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.919+0000 7fc3da7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:11.919 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.919+0000 7fc3da7fc640 10 client.?.objecter ms_handle_connect 0x7fc3e813f710 2024-03-25T19:31:11.919 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.919+0000 7fc3da7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:11.929 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.930+0000 7fc3da7fc640 10 client.?.objecter ms_handle_connect 0x7fc3e81430c0 2024-03-25T19:31:11.929 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.930+0000 7fc3da7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:11.930 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.930+0000 7fc3da7fc640 1 -- 172.21.15.83:0/3569736411 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3003475593 0 0) 0x7fc3e80704d0 con 0x7fc3e80a98f0 2024-03-25T19:31:11.930 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.930+0000 7fc3da7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:31:11.930 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.930+0000 7fc3da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:11.930 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.930+0000 7fc3da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:11.930 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.930+0000 7fc3da7fc640 10 monclient(hunting): my global_id is 4740 2024-03-25T19:31:11.930 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.930+0000 7fc3da7fc640 1 -- 172.21.15.83:0/3569736411 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3cc004eb0 con 0x7fc3e80a98f0 2024-03-25T19:31:11.939 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.940+0000 7fc3da7fc640 1 -- 172.21.15.83:0/3569736411 <== mon.1 v1:172.21.15.103:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1658775208 0 0) 0x7fc3e81df0d0 con 0x7fc3e81430c0 2024-03-25T19:31:11.939 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.940+0000 7fc3da7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:31:11.939 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.940+0000 7fc3da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:11.939 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.940+0000 7fc3da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:11.939 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.940+0000 7fc3da7fc640 10 monclient(hunting): my global_id is 4756 2024-03-25T19:31:11.940 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.940+0000 7fc3da7fc640 1 -- 172.21.15.83:0/3569736411 --> v1:172.21.15.103:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3e80704d0 con 0x7fc3e81430c0 2024-03-25T19:31:11.940 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.940+0000 7fc3da7fc640 1 -- 172.21.15.83:0/3569736411 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3753307177 0 0) 0x7fc3e806f490 con 0x7fc3e813f710 2024-03-25T19:31:11.940 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.940+0000 7fc3da7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:31:11.940 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.940+0000 7fc3da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:11.940 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.940+0000 7fc3da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:11.940 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.940+0000 7fc3da7fc640 10 monclient(hunting): my global_id is 4754 2024-03-25T19:31:11.940 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.940+0000 7fc3da7fc640 1 -- 172.21.15.83:0/3569736411 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3e81df0d0 con 0x7fc3e813f710 2024-03-25T19:31:11.947 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.947+0000 7fc3da7fc640 1 -- 172.21.15.83:0/3569736411 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 136278451 0 0) 0x7fc3e81df0d0 con 0x7fc3e813f710 2024-03-25T19:31:11.947 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.948+0000 7fc3da7fc640 1 -- 172.21.15.83:0/3569736411 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3e806f490 con 0x7fc3e813f710 2024-03-25T19:31:11.952 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.953+0000 7fc3da7fc640 1 -- 172.21.15.83:0/3569736411 <== mon.1 v1:172.21.15.103:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3365875653 0 0) 0x7fc3e80704d0 con 0x7fc3e81430c0 2024-03-25T19:31:11.952 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.953+0000 7fc3da7fc640 1 -- 172.21.15.83:0/3569736411 --> v1:172.21.15.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3e81df0d0 con 0x7fc3e81430c0 2024-03-25T19:31:11.953 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.953+0000 7fc3da7fc640 1 -- 172.21.15.83:0/3569736411 <== mon.1 v1:172.21.15.103:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fc3e00018f0 con 0x7fc3e81430c0 2024-03-25T19:31:11.953 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.953+0000 7fc3da7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:11.953 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.953+0000 7fc3da7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:31:11.953 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.953+0000 7fc3da7fc640 10 monclient(hunting): dump: 2024-03-25T19:31:11.953 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:31:11.954 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:31:11.954 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:31:11.954 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:31:11.954 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:11.954 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:31:11.954 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:31:11.954 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:31:11.954 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:31:11.954 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:31:11.954 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.953+0000 7fc3da7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:31:11.954 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.953+0000 7fc3da7fc640 1 -- 172.21.15.83:0/3569736411 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4041474644 0 0) 0x7fc3cc004eb0 con 0x7fc3e80a98f0 2024-03-25T19:31:11.954 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.953+0000 7fc3da7fc640 1 -- 172.21.15.83:0/3569736411 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3e80704d0 con 0x7fc3e80a98f0 2024-03-25T19:31:11.954 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.953+0000 7fc3da7fc640 1 -- 172.21.15.83:0/3569736411 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fc3dc001b00 con 0x7fc3e80a98f0 2024-03-25T19:31:11.954 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.953+0000 7fc3da7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:11.954 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.953+0000 7fc3da7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:31:11.955 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.953+0000 7fc3da7fc640 10 monclient(hunting): dump: 2024-03-25T19:31:11.955 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:31:11.955 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:31:11.955 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:31:11.955 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:31:11.955 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:11.955 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:31:11.955 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:31:11.955 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:31:11.955 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:31:11.955 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:31:11.956 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.955+0000 7fc3da7fc640 1 -- 172.21.15.83:0/3569736411 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fc3e40019f0 con 0x7fc3e813f710 2024-03-25T19:31:11.956 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.955+0000 7fc3da7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:11.956 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.955+0000 7fc3da7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:31:11.956 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.955+0000 7fc3da7fc640 10 monclient(hunting): dump: 2024-03-25T19:31:11.956 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:31:11.956 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:31:11.956 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:31:11.956 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:31:11.956 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:11.956 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:31:11.956 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:31:11.956 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:31:11.956 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:31:11.957 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:31:11.960 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.959+0000 7fc3da7fc640 1 -- 172.21.15.83:0/3569736411 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 997964288 0 0) 0x7fc3e806f490 con 0x7fc3e813f710 2024-03-25T19:31:11.960 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.959+0000 7fc3da7fc640 1 -- 172.21.15.83:0/3569736411 >> v1:172.21.15.103:6789/0 conn(0x7fc3e81430c0 legacy=0x7fc3e8138ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:11.960 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.959+0000 7fc3da7fc640 1 -- 172.21.15.83:0/3569736411 >> v1:172.21.15.83:6789/0 conn(0x7fc3e80a98f0 legacy=0x7fc3e806ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:11.960 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.959+0000 7fc3da7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:31:11.960 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.959+0000 7fc3da7fc640 1 monclient: found mon.c 2024-03-25T19:31:11.960 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.959+0000 7fc3da7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:11.960 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.959+0000 7fc3da7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:31:11.961 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.959+0000 7fc3da7fc640 1 -- 172.21.15.83:0/3569736411 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e81dfcc0 con 0x7fc3e813f710 2024-03-25T19:31:11.961 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.959+0000 7fc3da7fc640 10 monclient: _finish_auth 0 2024-03-25T19:31:11.961 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.959+0000 7fc3da7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:31:11.961 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.959+0000 7fc3da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:11.961 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.959+0000 7fc3efb6d640 5 monclient: authenticate success, global_id 4754 2024-03-25T19:31:11.961 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.959+0000 7fc3efb6d640 10 monclient: _renew_subs 2024-03-25T19:31:11.961 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.959+0000 7fc3efb6d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:31:11.961 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.959+0000 7fc3efb6d640 1 -- 172.21.15.83:0/3569736411 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3e806e0e0 con 0x7fc3e813f710 2024-03-25T19:31:11.961 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.959+0000 7fc3efb6d640 10 client.4754.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:31:11.961 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.959+0000 7fc3efb6d640 10 monclient: _renew_subs 2024-03-25T19:31:11.961 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.959+0000 7fc3efb6d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:31:11.961 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.959+0000 7fc3efb6d640 1 -- 172.21.15.83:0/3569736411 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3e8070db0 con 0x7fc3e813f710 2024-03-25T19:31:11.961 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.959+0000 7fc3efb6d640 1 librados: init done 2024-03-25T19:31:11.961 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.960+0000 7fc3efb6d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:11.961 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.960+0000 7fc3efb6d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:11.961 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.960+0000 7fc3efb6d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:31:11.962 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:11.960+0000 7fc3efb6d640 1 -- 172.21.15.83:0/3569736411 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3e81def30 con 0x7fc3e813f710 2024-03-25T19:31:12.279 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:12.279+0000 7fc3da7fc640 1 -- 172.21.15.83:0/3569736411 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3e40035f0 con 0x7fc3e813f710 2024-03-25T19:31:12.279 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:12.279+0000 7fc3da7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:12.279 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:12.279+0000 7fc3da7fc640 1 -- 172.21.15.83:0/3569736411 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3931250204 0 0) 0x7fc3e40016d0 con 0x7fc3e813f710 2024-03-25T19:31:12.279 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:12.279+0000 7fc3da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:12.279 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:12.279+0000 7fc3da7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:31:12.279 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:12.279+0000 7fc3da7fc640 10 monclient: dump: 2024-03-25T19:31:12.279 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T19:31:12.279 INFO:teuthology.misc.health.smithi083.stderr:fsid e1ed2130-f46c-41bd-8514-ee278101259f 2024-03-25T19:31:12.279 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T19:18:45.215859+0000 2024-03-25T19:31:12.279 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T19:18:45.215859+0000 2024-03-25T19:31:12.279 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:12.279 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 3 2024-03-25T19:31:12.280 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.a 2024-03-25T19:31:12.280 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.103:6789/0 mon.b 2024-03-25T19:31:12.280 INFO:teuthology.misc.health.smithi083.stderr:2: v1:172.21.15.83:6790/0 mon.c 2024-03-25T19:31:12.280 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T19:31:12.280 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:12.279+0000 7fc3dbfff640 4 set_mon_vals no callback set 2024-03-25T19:31:12.280 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:12.279+0000 7fc3da7fc640 1 -- 172.21.15.83:0/3569736411 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3415740686 0 0) 0x7fc3e4003920 con 0x7fc3e813f710 2024-03-25T19:31:12.280 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:12.280+0000 7fc3ed909640 10 monclient: get_auth_request con 0x7fc3cc06e0d0 auth_method 0 2024-03-25T19:31:12.281 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:12.280+0000 7fc3da7fc640 10 client.4754.objecter ms_handle_connect 0x7fc3cc06e0d0 2024-03-25T19:31:12.284 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:12.284+0000 7fc3da7fc640 1 -- 172.21.15.83:0/3569736411 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2659774754 0 0) 0x7fc3e4054590 con 0x7fc3e813f710 2024-03-25T19:31:12.284 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:12.284+0000 7fc3da7fc640 10 client.4754.objecter ms_dispatch 0x7fc3e8000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:31:12.284 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:12.284+0000 7fc3da7fc640 3 client.4754.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:31:12.285 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:12.284+0000 7fc3da7fc640 3 client.4754.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:31:12.285 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:12.284+0000 7fc3da7fc640 20 client.4754.objecter dump_active .. 0 homeless 2024-03-25T19:31:12.285 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:12.284+0000 7fc3da7fc640 1 -- 172.21.15.83:0/3569736411 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc3e404fed0 con 0x7fc3e813f710 2024-03-25T19:31:12.285 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:12.284+0000 7fc3da7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:12.285 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:12.284+0000 7fc3da7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:31:12.524 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:12.525+0000 7fc3efb6d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:12.525 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:12.525+0000 7fc3efb6d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:12.525 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:12.525+0000 7fc3efb6d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.83:6790/0 2024-03-25T19:31:12.525 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:12.525+0000 7fc3efb6d640 1 -- 172.21.15.83:0/3569736411 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3e806e3e0 con 0x7fc3e813f710 2024-03-25T19:31:12.559 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:12.560+0000 7fc3da7fc640 1 -- 172.21.15.83:0/3569736411 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1070525753) 0x7fc3e4051fd0 con 0x7fc3e813f710 2024-03-25T19:31:12.559 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:12.560+0000 7fc3da7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:12.559 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:12.560+0000 7fc3da7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:31:12.560 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T19:31:12.560 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":11,"fsid":"e1ed2130-f46c-41bd-8514-ee278101259f","created":"2024-03-25T19:19:45.318410+0000","modified":"2024-03-25T19:29:48.933916+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:48.933916+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79cfe51b-0994-40c6-b556-3e09b3e0fbf7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b51f462f-9495-4b42-94a1-cfa31e1db349","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"467ce07c-8165-4299-90b2-274bc771cff5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a412567e-e1ad-4d42-8e3a-344cbe4fb874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8354d57d-5c88-4259-995b-94c71f146960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"801edd90-2540-4d68-aefd-f9766dff347c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"56c6dbb7-33a7-4af1-a6ab-26c25e08f14c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"612c69a8-2abe-482c-b05c-8909132855ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:31:12.562 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:12.562+0000 7fc3efb6d640 10 librados: watch_flush enter 2024-03-25T19:31:12.562 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:12.562+0000 7fc3efb6d640 10 librados: watch_flush exit 2024-03-25T19:31:12.562 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:12.562+0000 7fc3efb6d640 20 client.4754.objecter shutdown clearing up homeless session... 2024-03-25T19:31:12.562 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:12.562+0000 7fc3efb6d640 10 client.4754.objecter successfully canceled tick 2024-03-25T19:31:12.562 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:12.562+0000 7fc3efb6d640 1 -- 172.21.15.83:0/3569736411 >> v1:172.21.15.103:6800/345072327 conn(0x7fc3cc06e0d0 legacy=0x7fc3cc070590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:12.562 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:12.562+0000 7fc3efb6d640 10 monclient: shutdown 2024-03-25T19:31:12.563 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:12.562+0000 7fc3efb6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:12.563 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:12.562+0000 7fc3efb6d640 1 -- 172.21.15.83:0/3569736411 >> v1:172.21.15.83:6790/0 conn(0x7fc3e813f710 legacy=0x7fc3e81df5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:12.563 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:12.562+0000 7fc3efb6d640 1 -- 172.21.15.83:0/3569736411 shutdown_connections 2024-03-25T19:31:12.563 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:12.562+0000 7fc3efb6d640 1 -- 172.21.15.83:0/3569736411 >> 172.21.15.83:0/3569736411 conn(0x7fc3e80b7ea0 msgr2=0x7fc3e814e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:12.563 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:12.562+0000 7fc3efb6d640 1 -- 172.21.15.83:0/3569736411 shutdown_connections 2024-03-25T19:31:12.563 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:12.562+0000 7fc3efb6d640 1 -- 172.21.15.83:0/3569736411 wait complete. 2024-03-25T19:31:12.563 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T19:31:12.562+0000 7fc3efb6d640 1 librados: shutdown 2024-03-25T19:31:12.574 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:31:12.574 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-25T19:31:12.576 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-25T19:31:12.576 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-25T19:31:12.576 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:31:14.493 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:11:56.420 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:14.514 INFO:tasks.ceph.osd.0.smithi083.stderr:==00:00:11:24.010 35986== 2024-03-25T19:31:14.514 INFO:tasks.ceph.osd.0.smithi083.stderr:==00:00:11:24.010 35986== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:31:14.542 INFO:tasks.ceph.osd.0.smithi083.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:31:14.543 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:11:56.471 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:14.579 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:11:56.507 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:14.598 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:11:56.508 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:14.601 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:11:56.512 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:14.604 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:11:56.531 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:14.614 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:11:56.542 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:14.625 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:11:56.534 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:14.644 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:11:56.571 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:18.678 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:31:18.678 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi083 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0" 2024-03-25T19:31:18.679 INFO:tasks.ceph.osd.0:Stopped 2024-03-25T19:31:18.679 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-25T19:31:18.680 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:31:20.533 INFO:tasks.ceph.osd.1.smithi083.stderr:==00:00:11:30.028 35987== 2024-03-25T19:31:20.533 INFO:tasks.ceph.osd.1.smithi083.stderr:==00:00:11:30.028 35987== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:31:20.571 INFO:tasks.ceph.osd.1.smithi083.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:31:24.781 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:31:24.781 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi083 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1" 2024-03-25T19:31:24.782 INFO:tasks.ceph.osd.1:Stopped 2024-03-25T19:31:24.782 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-25T19:31:24.782 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:31:26.825 INFO:tasks.ceph.osd.2.smithi083.stderr:==00:00:11:36.317 35988== 2024-03-25T19:31:26.825 INFO:tasks.ceph.osd.2.smithi083.stderr:==00:00:11:36.317 35988== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:31:26.865 INFO:tasks.ceph.osd.2.smithi083.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:31:30.884 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:31:30.884 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi083 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2" 2024-03-25T19:31:30.885 INFO:tasks.ceph.osd.2:Stopped 2024-03-25T19:31:30.885 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-25T19:31:30.885 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:31:32.770 INFO:tasks.ceph.osd.3.smithi083.stderr:==00:00:11:42.255 35989== 2024-03-25T19:31:32.771 INFO:tasks.ceph.osd.3.smithi083.stderr:==00:00:11:42.255 35989== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:31:32.811 INFO:tasks.ceph.osd.3.smithi083.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:31:34.583 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:12:16.511 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:34.591 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:12:16.518 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:34.610 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:12:16.537 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:34.641 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:12:16.551 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:34.643 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:12:16.570 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:34.647 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:12:16.558 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:34.648 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:12:16.575 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:34.672 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:12:16.599 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:34.677 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:12:16.587 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:36.986 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:31:36.987 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi083 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3" 2024-03-25T19:31:36.987 INFO:tasks.ceph.osd.3:Stopped 2024-03-25T19:31:36.987 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-25T19:31:36.987 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:31:38.891 INFO:tasks.ceph.osd.4.smithi103.stderr:==00:00:11:48.400 35632== 2024-03-25T19:31:38.891 INFO:tasks.ceph.osd.4.smithi103.stderr:==00:00:11:48.400 35632== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:31:38.931 INFO:tasks.ceph.osd.4.smithi103.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:31:43.088 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:31:43.089 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi103 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4" 2024-03-25T19:31:43.089 INFO:tasks.ceph.osd.4:Stopped 2024-03-25T19:31:43.089 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-25T19:31:43.090 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:31:44.902 INFO:tasks.ceph.osd.5.smithi103.stderr:==00:00:11:54.395 35633== 2024-03-25T19:31:44.902 INFO:tasks.ceph.osd.5.smithi103.stderr:==00:00:11:54.395 35633== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:31:44.942 INFO:tasks.ceph.osd.5.smithi103.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:31:49.190 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:31:49.191 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi103 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5" 2024-03-25T19:31:49.191 INFO:tasks.ceph.osd.5:Stopped 2024-03-25T19:31:49.191 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-25T19:31:49.191 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:31:51.012 INFO:tasks.ceph.osd.6.smithi103.stderr:==00:00:12:00.496 35634== 2024-03-25T19:31:51.012 INFO:tasks.ceph.osd.6.smithi103.stderr:==00:00:12:00.496 35634== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:31:51.049 INFO:tasks.ceph.osd.6.smithi103.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:31:54.623 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:12:36.551 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:54.627 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:12:36.555 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:54.645 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:12:36.572 35673== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:54.671 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:12:36.581 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:54.675 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:12:36.603 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:54.676 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:12:36.586 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:54.679 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:12:36.607 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:54.693 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:12:36.603 35089== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:54.694 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:12:36.622 35675== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:55.293 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:31:55.294 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi103 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6" 2024-03-25T19:31:55.294 INFO:tasks.ceph.osd.6:Stopped 2024-03-25T19:31:55.295 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-25T19:31:55.295 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:31:56.838 INFO:tasks.ceph.osd.7.smithi103.stderr:==00:00:12:06.324 35635== 2024-03-25T19:31:56.838 INFO:tasks.ceph.osd.7.smithi103.stderr:==00:00:12:06.324 35635== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:31:56.869 INFO:tasks.ceph.osd.7.smithi103.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:32:01.397 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:32:01.397 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi103 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7" 2024-03-25T19:32:01.397 INFO:tasks.ceph.osd.7:Stopped 2024-03-25T19:32:01.397 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-25T19:32:01.398 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-25T19:32:01.398 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:32:01.422 INFO:tasks.ceph.mgr.y:Stopped 2024-03-25T19:32:01.422 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-25T19:32:01.422 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:32:01.423 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-03-25T19:32:01.424+0000 7f029d53a640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i x (PID: 35087) UID: 0 2024-03-25T19:32:01.423 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-03-25T19:32:01.424+0000 7f029d53a640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-25T19:32:01.452 INFO:tasks.ceph.mgr.x:Stopped 2024-03-25T19:32:01.452 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-25T19:32:01.452 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-25T19:32:01.452 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:32:01.477 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-03-25T19:32:01.467+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 35640) UID: 0 2024-03-25T19:32:01.478 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-03-25T19:32:01.468+0000 9f86640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-25T19:32:01.880 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:12:43.807 35673== 2024-03-25T19:32:01.880 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:12:43.807 35673== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:32:01.889 INFO:tasks.ceph.mon.a.smithi083.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:32:07.554 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:32:07.555 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi083 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-03-25T19:32:07.555 INFO:tasks.ceph.mon.a:Stopped 2024-03-25T19:32:07.555 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-25T19:32:07.555 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:32:07.571 INFO:tasks.ceph.mon.c.smithi083.stderr:2024-03-25T19:32:07.560+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 35642) UID: 0 2024-03-25T19:32:07.572 INFO:tasks.ceph.mon.c.smithi083.stderr:2024-03-25T19:32:07.562+0000 9f86640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-03-25T19:34:31.538 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:15:13.465 35675== 2024-03-25T19:34:31.538 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:15:13.465 35675== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:34:31.548 INFO:tasks.ceph.mon.c.smithi083.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:34:33.991 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:34:33.991 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi083 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-03-25T19:34:33.992 INFO:tasks.ceph.mon.c:Stopped 2024-03-25T19:34:33.993 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-25T19:34:33.993 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:34:34.009 INFO:tasks.ceph.mon.b.smithi103.stderr:2024-03-25T19:34:33.998+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 35085) UID: 0 2024-03-25T19:34:34.009 INFO:tasks.ceph.mon.b.smithi103.stderr:2024-03-25T19:34:33.999+0000 9f86640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2024-03-25T19:36:52.452 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:17:34.361 35089== 2024-03-25T19:36:52.452 INFO:tasks.ceph.mon.b.smithi103.stderr:==00:00:17:34.361 35089== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:36:52.460 INFO:tasks.ceph.mon.b.smithi103.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:36:54.327 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:36:54.327 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi103 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-03-25T19:36:54.328 INFO:tasks.ceph.mon.b:Stopped 2024-03-25T19:36:54.328 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-25T19:36:54.329 DEBUG:teuthology.orchestra.run.smithi083:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-25T19:36:54.375 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi083.front.sepia.ceph.com 2024-03-25T19:36:54.375 DEBUG:teuthology.orchestra.run.smithi083:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-25T19:36:54.433 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi083.front.sepia.ceph.com 2024-03-25T19:36:54.433 DEBUG:teuthology.orchestra.run.smithi083:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-25T19:36:54.491 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi083.front.sepia.ceph.com 2024-03-25T19:36:54.491 DEBUG:teuthology.orchestra.run.smithi083:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-25T19:36:54.548 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi083.front.sepia.ceph.com 2024-03-25T19:36:54.548 DEBUG:teuthology.orchestra.run.smithi083:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-25T19:36:54.598 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi103.front.sepia.ceph.com 2024-03-25T19:36:54.598 DEBUG:teuthology.orchestra.run.smithi103:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-25T19:36:54.899 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi103.front.sepia.ceph.com 2024-03-25T19:36:54.899 DEBUG:teuthology.orchestra.run.smithi103:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-25T19:36:54.978 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi103.front.sepia.ceph.com 2024-03-25T19:36:54.979 DEBUG:teuthology.orchestra.run.smithi103:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-25T19:36:55.020 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi103.front.sepia.ceph.com 2024-03-25T19:36:55.020 DEBUG:teuthology.orchestra.run.smithi103:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-25T19:36:55.123 INFO:tasks.ceph:Archiving mon data... 2024-03-25T19:36:55.124 DEBUG:teuthology.misc:Transferring archived files from smithi083:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621304/data/mon.a.tgz 2024-03-25T19:36:55.124 DEBUG:teuthology.orchestra.run.smithi083:> mktemp 2024-03-25T19:36:55.141 INFO:teuthology.orchestra.run.smithi083.stdout:/tmp/tmp.3u9ZKi7unO 2024-03-25T19:36:55.141 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.3u9ZKi7unO 2024-03-25T19:36:55.262 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 0666 /tmp/tmp.3u9ZKi7unO 2024-03-25T19:36:55.351 DEBUG:teuthology.orchestra.remote:smithi083:/tmp/tmp.3u9ZKi7unO is 132KB 2024-03-25T19:36:55.404 DEBUG:teuthology.orchestra.run.smithi083:> rm -fr /tmp/tmp.3u9ZKi7unO 2024-03-25T19:36:55.419 DEBUG:teuthology.misc:Transferring archived files from smithi083:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621304/data/mon.c.tgz 2024-03-25T19:36:55.419 DEBUG:teuthology.orchestra.run.smithi083:> mktemp 2024-03-25T19:36:55.474 INFO:teuthology.orchestra.run.smithi083.stdout:/tmp/tmp.CwSLqQsmy2 2024-03-25T19:36:55.474 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.CwSLqQsmy2 2024-03-25T19:36:55.562 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 0666 /tmp/tmp.CwSLqQsmy2 2024-03-25T19:36:55.646 DEBUG:teuthology.orchestra.remote:smithi083:/tmp/tmp.CwSLqQsmy2 is 132KB 2024-03-25T19:36:55.668 DEBUG:teuthology.orchestra.run.smithi083:> rm -fr /tmp/tmp.CwSLqQsmy2 2024-03-25T19:36:55.682 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621304/data/mon.b.tgz 2024-03-25T19:36:55.682 DEBUG:teuthology.orchestra.run.smithi103:> mktemp 2024-03-25T19:36:55.699 INFO:teuthology.orchestra.run.smithi103.stdout:/tmp/tmp.SvXDk0Y76a 2024-03-25T19:36:55.699 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.SvXDk0Y76a 2024-03-25T19:36:55.806 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0666 /tmp/tmp.SvXDk0Y76a 2024-03-25T19:36:55.896 DEBUG:teuthology.orchestra.remote:smithi103:/tmp/tmp.SvXDk0Y76a is 133KB 2024-03-25T19:36:55.920 DEBUG:teuthology.orchestra.run.smithi103:> rm -fr /tmp/tmp.SvXDk0Y76a 2024-03-25T19:36:55.934 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-25T19:36:55.934 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T19:36:55.937 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T19:36:56.259 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T19:36:56.526 INFO:tasks.ceph:Archiving crash dumps... 2024-03-25T19:36:56.528 DEBUG:teuthology.misc:Transferring archived files from smithi083:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621304/remote/smithi083/crash 2024-03-25T19:36:56.529 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T19:36:56.556 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621304/remote/smithi103/crash 2024-03-25T19:36:56.557 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T19:36:56.583 INFO:tasks.ceph:Compressing logs... 2024-03-25T19:36:56.583 DEBUG:teuthology.orchestra.run.smithi083:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T19:36:56.601 DEBUG:teuthology.orchestra.run.smithi103:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T19:36:56.624 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40305.log 2024-03-25T19:36:56.625 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42450.log 2024-03-25T19:36:56.625 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40890.log 2024-03-25T19:36:56.665 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-25T19:36:56.665 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-03-25T19:36:56.665 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-03-25T19:36:56.665 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-03-25T19:36:56.665 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/valgrind/osd.5.log: 72.1% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-03-25T19:36:56.665 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-25T19:36:56.665 INFO:teuthology.orchestra.run.smithi103.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-03-25T19:36:56.666 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/valgrind/osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-03-25T19:36:56.666 INFO:teuthology.orchestra.run.smithi103.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-03-25T19:36:56.666 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35442.log 2024-03-25T19:36:56.666 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/valgrind/mon.b.log: /var/log/ceph/valgrind/osd.4.log: 69.6% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-25T19:36:56.666 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35362.log 2024-03-25T19:36:56.666 INFO:teuthology.orchestra.run.smithi103.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-03-25T19:36:56.666 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.35442.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T19:36:56.666 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.35362.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-25T19:36:56.666 INFO:teuthology.orchestra.run.smithi103.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35442.log.gz 2024-03-25T19:36:56.666 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-25T19:36:56.666 INFO:teuthology.orchestra.run.smithi103.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35362.log.gz 2024-03-25T19:36:56.666 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-osd.4.log: gzip 87.9% -- replaced with /var/log/ceph/ceph.audit.log.gz -5 2024-03-25T19:36:56.666 INFO:teuthology.orchestra.run.smithi103.stderr: --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-25T19:36:56.666 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35533.log 2024-03-25T19:36:56.667 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-osd.7.log: /var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35322.log 2024-03-25T19:36:56.667 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.35533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35242.log 2024-03-25T19:36:56.667 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.35322.log: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-25T19:36:56.667 INFO:teuthology.orchestra.run.smithi103.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35533.log.gz 2024-03-25T19:36:56.667 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.35242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35282.log 2024-03-25T19:36:56.667 INFO:teuthology.orchestra.run.smithi103.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35322.log.gz 2024-03-25T19:36:56.667 INFO:teuthology.orchestra.run.smithi103.stderr:gzip/var/log/ceph/ceph-mgr.x.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.35484.log 2024-03-25T19:36:56.667 INFO:teuthology.orchestra.run.smithi103.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35242.log.gz 2024-03-25T19:36:56.667 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35402.log 2024-03-25T19:36:56.667 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.35282.log: /var/log/ceph/ceph-client.admin.35484.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T19:36:56.667 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.35402.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35484.log.gz 2024-03-25T19:36:56.667 INFO:teuthology.orchestra.run.smithi103.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35282.log.gz 2024-03-25T19:36:56.667 INFO:teuthology.orchestra.run.smithi103.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35402.log.gz 2024-03-25T19:36:56.667 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-25T19:36:56.667 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-mon.b.log: 87.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T19:36:56.675 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41046.log 2024-03-25T19:36:56.675 INFO:teuthology.orchestra.run.smithi083.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40305.log.gz 2024-03-25T19:36:56.675 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40890.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-25T19:36:56.676 INFO:teuthology.orchestra.run.smithi083.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.42450.log.gz 2024-03-25T19:36:56.676 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41046.log: gzip -5 --verbose -- 86.2% /var/log/ceph/ceph-client.admin.40383.log -- replaced with /var/log/ceph/ceph-client.admin.40890.log.gz 2024-03-25T19:36:56.676 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T19:36:56.676 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35682.log 2024-03-25T19:36:56.676 INFO:teuthology.orchestra.run.smithi083.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41046.log.gz 2024-03-25T19:36:56.676 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39560.log 2024-03-25T19:36:56.676 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39601.log 2024-03-25T19:36:56.676 INFO:teuthology.orchestra.run.smithi083.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40383.log.gz 2024-03-25T19:36:56.676 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.39560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41397.log 2024-03-25T19:36:56.676 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.39601.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39560.log.gz 2024-03-25T19:36:56.676 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41904.log 2024-03-25T19:36:56.676 INFO:teuthology.orchestra.run.smithi083.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.35682.log.gz 2024-03-25T19:36:56.676 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42294.log 2024-03-25T19:36:56.677 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41397.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39601.log.gz 2024-03-25T19:36:56.677 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40149.log 2024-03-25T19:36:56.677 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42294.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41397.log.gz 2024-03-25T19:36:56.677 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-25T19:36:56.677 INFO:teuthology.orchestra.run.smithi083.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41904.log.gz 2024-03-25T19:36:56.677 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40149.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-03-25T19:36:56.677 INFO:teuthology.orchestra.run.smithi083.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42294.log.gz 2024-03-25T19:36:56.677 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-03-25T19:36:56.677 INFO:teuthology.orchestra.run.smithi083.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40149.log.gz 2024-03-25T19:36:56.677 INFO:teuthology.orchestra.run.smithi083.stderr: 69.5% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-25T19:36:56.677 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-03-25T19:36:56.677 INFO:teuthology.orchestra.run.smithi083.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-03-25T19:36:56.677 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-25T19:36:56.677 INFO:teuthology.orchestra.run.smithi083.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-03-25T19:36:56.677 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-03-25T19:36:56.678 INFO:teuthology.orchestra.run.smithi083.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-03-25T19:36:56.678 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41202.log 2024-03-25T19:36:56.678 INFO:teuthology.orchestra.run.smithi083.stderr: 69.5% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-25T19:36:56.678 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42503.log 2024-03-25T19:36:56.678 INFO:teuthology.orchestra.run.smithi083.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-03-25T19:36:56.678 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41007.log 2024-03-25T19:36:56.678 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35985.log 2024-03-25T19:36:56.678 INFO:teuthology.orchestra.run.smithi083.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41202.log.gz 2024-03-25T19:36:56.678 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40266.log 2024-03-25T19:36:56.678 INFO:teuthology.orchestra.run.smithi083.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.42503.log.gz 2024-03-25T19:36:56.678 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42661.log 2024-03-25T19:36:56.678 INFO:teuthology.orchestra.run.smithi083.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41007.log.gz 2024-03-25T19:36:56.678 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40266.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.35985.log.gz 2024-03-25T19:36:56.678 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40578.log 2024-03-25T19:36:56.678 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40422.log 2024-03-25T19:36:56.679 INFO:teuthology.orchestra.run.smithi083.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40266.log.gz 2024-03-25T19:36:56.679 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40734.log 2024-03-25T19:36:56.679 INFO:teuthology.orchestra.run.smithi083.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.42661.log.gz 2024-03-25T19:36:56.679 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40929.log 2024-03-25T19:36:56.679 INFO:teuthology.orchestra.run.smithi083.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40578.log.gz 2024-03-25T19:36:56.679 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5/var/log/ceph/ceph-client.admin.40734.log: --verbose -- /var/log/ceph/ceph-client.admin.41358.log 2024-03-25T19:36:56.679 INFO:teuthology.orchestra.run.smithi083.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40422.log.gz 2024-03-25T19:36:56.679 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41553.log 2024-03-25T19:36:56.679 INFO:teuthology.orchestra.run.smithi083.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40734.log.gz 2024-03-25T19:36:56.679 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38018.log 2024-03-25T19:36:56.679 INFO:teuthology.orchestra.run.smithi083.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40929.log.gz 2024-03-25T19:36:56.679 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40500.log 2024-03-25T19:36:56.679 INFO:teuthology.orchestra.run.smithi083.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41358.log.gz 2024-03-25T19:36:56.679 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.38018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42372.log 2024-03-25T19:36:56.680 INFO:teuthology.orchestra.run.smithi083.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41553.log.gz 2024-03-25T19:36:56.680 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40500.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38018.log.gz 2024-03-25T19:36:56.680 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42411.log 2024-03-25T19:36:56.680 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39797.log 2024-03-25T19:36:56.680 INFO:teuthology.orchestra.run.smithi083.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40500.log.gz 2024-03-25T19:36:56.680 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42411.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T19:36:56.680 INFO:teuthology.orchestra.run.smithi083.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.42372.log.gz 2024-03-25T19:36:56.680 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.39797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41475.log 2024-03-25T19:36:56.680 INFO:teuthology.orchestra.run.smithi083.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.42411.log.gz 2024-03-25T19:36:56.680 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph.audit.log: 86.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.39797.log.gz 2024-03-25T19:36:56.680 INFO:teuthology.orchestra.run.smithi083.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40773.log 2024-03-25T19:36:56.680 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42099.log 2024-03-25T19:36:56.680 INFO:teuthology.orchestra.run.smithi083.stderr: 93.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T19:36:56.680 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42021.log 2024-03-25T19:36:56.680 INFO:teuthology.orchestra.run.smithi083.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41475.log.gz 2024-03-25T19:36:56.681 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40070.log 2024-03-25T19:36:56.681 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42099.log: /var/log/ceph/ceph-client.admin.42021.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40773.log.gz 2024-03-25T19:36:56.681 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39680.log 2024-03-25T19:36:56.681 INFO:teuthology.orchestra.run.smithi083.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.42099.log.gz 2024-03-25T19:36:56.681 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40070.log: 86.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.42021.log.gz -5 2024-03-25T19:36:56.681 INFO:teuthology.orchestra.run.smithi083.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40227.log 2024-03-25T19:36:56.681 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42700.log 2024-03-25T19:36:56.681 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.39680.log: /var/log/ceph/ceph-client.admin.40227.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40070.log.gz 2024-03-25T19:36:56.681 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42060.log 2024-03-25T19:36:56.681 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42700.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39680.log.gz 2024-03-25T19:36:56.681 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41943.log 2024-03-25T19:36:56.681 INFO:teuthology.orchestra.run.smithi083.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40227.log.gz 2024-03-25T19:36:56.681 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41631.log/var/log/ceph/ceph-client.admin.42060.log: 2024-03-25T19:36:56.681 INFO:teuthology.orchestra.run.smithi083.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.42700.log.gz 2024-03-25T19:36:56.681 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40695.log 2024-03-25T19:36:56.682 INFO:teuthology.orchestra.run.smithi083.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.42060.log.gz 2024-03-25T19:36:56.682 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36012.log 2024-03-25T19:36:56.682 INFO:teuthology.orchestra.run.smithi083.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41943.log.gz 2024-03-25T19:36:56.682 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39521.log 2024-03-25T19:36:56.682 INFO:teuthology.orchestra.run.smithi083.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41631.log.gz 2024-03-25T19:36:56.682 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.36012.log: gzip -5 87.2% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.40695.log.gz 2024-03-25T19:36:56.682 INFO:teuthology.orchestra.run.smithi083.stderr: /var/log/ceph/ceph-osd.1.log 2024-03-25T19:36:56.682 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.39521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36147.log 2024-03-25T19:36:56.682 INFO:teuthology.orchestra.run.smithi083.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.36012.log.gz 2024-03-25T19:36:56.682 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41241.log 2024-03-25T19:36:56.682 INFO:teuthology.orchestra.run.smithi083.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39521.log.gz 2024-03-25T19:36:56.682 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.36147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40968.log 2024-03-25T19:36:56.682 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41241.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.36147.log.gz 2024-03-25T19:36:56.682 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41280.log 2024-03-25T19:36:56.683 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39875.log 2024-03-25T19:36:56.683 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40968.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41241.log.gz 2024-03-25T19:36:56.683 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40851.log 2024-03-25T19:36:56.683 INFO:teuthology.orchestra.run.smithi083.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40968.log.gz 2024-03-25T19:36:56.683 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.39875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39758.log 2024-03-25T19:36:56.683 INFO:teuthology.orchestra.run.smithi083.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41280.log.gz 2024-03-25T19:36:56.683 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41085.log 2024-03-25T19:36:56.683 INFO:teuthology.orchestra.run.smithi083.stderr: 87.0%/var/log/ceph/ceph-client.admin.39758.log: -- replaced with /var/log/ceph/ceph-client.admin.39875.log.gz 2024-03-25T19:36:56.683 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42138.log 2024-03-25T19:36:56.683 INFO:teuthology.orchestra.run.smithi083.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40851.log.gz 2024-03-25T19:36:56.683 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41085.log: 86.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.39758.log.gz -5 2024-03-25T19:36:56.683 INFO:teuthology.orchestra.run.smithi083.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41709.log 2024-03-25T19:36:56.683 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41124.log 2024-03-25T19:36:56.683 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42138.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41085.log.gz 2024-03-25T19:36:56.683 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39299.log 2024-03-25T19:36:56.683 INFO:teuthology.orchestra.run.smithi083.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.42138.log.gz 2024-03-25T19:36:56.684 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41163.log 2024-03-25T19:36:56.684 INFO:teuthology.orchestra.run.smithi083.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41709.log.gz 2024-03-25T19:36:56.684 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.39299.log: 86.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.41124.log.gz -5 2024-03-25T19:36:56.684 INFO:teuthology.orchestra.run.smithi083.stderr: --verbose -- /var/log/ceph/ceph-client.admin.42739.log 2024-03-25T19:36:56.684 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-25T19:36:56.684 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41163.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39299.log.gz 2024-03-25T19:36:56.684 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40031.log 2024-03-25T19:36:56.684 INFO:teuthology.orchestra.run.smithi083.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41163.log.gz 2024-03-25T19:36:56.684 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41670.log 2024-03-25T19:36:56.684 INFO:teuthology.orchestra.run.smithi083.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42739.log.gz 2024-03-25T19:36:56.684 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42817.log 2024-03-25T19:36:56.684 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41514.log 2024-03-25T19:36:56.684 INFO:teuthology.orchestra.run.smithi083.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40031.log.gz 2024-03-25T19:36:56.684 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39640.log 2024-03-25T19:36:56.684 INFO:teuthology.orchestra.run.smithi083.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41670.log.gz 2024-03-25T19:36:56.684 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41787.log 2024-03-25T19:36:56.685 INFO:teuthology.orchestra.run.smithi083.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42817.log.gz 2024-03-25T19:36:56.685 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.39640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41592.log 2024-03-25T19:36:56.685 INFO:teuthology.orchestra.run.smithi083.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41514.log.gz 2024-03-25T19:36:56.685 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41787.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39640.log.gz 2024-03-25T19:36:56.685 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41319.log 2024-03-25T19:36:56.685 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39992.log 2024-03-25T19:36:56.685 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41592.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41787.log.gz 2024-03-25T19:36:56.685 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41319.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29831.log 2024-03-25T19:36:56.685 INFO:teuthology.orchestra.run.smithi083.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41592.log.gz 2024-03-25T19:36:56.685 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.39992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40188.log 2024-03-25T19:36:56.685 INFO:teuthology.orchestra.run.smithi083.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41319.log.gz 2024-03-25T19:36:56.685 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph.tmp-client.admin.29831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41748.log 2024-03-25T19:36:56.685 INFO:teuthology.orchestra.run.smithi083.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29831.log.gz 2024-03-25T19:36:56.685 INFO:teuthology.orchestra.run.smithi083.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39992.log.gz 2024-03-25T19:36:56.685 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41436.log 2024-03-25T19:36:56.686 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40539.log 2024-03-25T19:36:56.686 INFO:teuthology.orchestra.run.smithi083.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40188.log.gz 2024-03-25T19:36:56.686 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40812.log 2024-03-25T19:36:56.686 INFO:teuthology.orchestra.run.smithi083.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41748.log.gz 2024-03-25T19:36:56.686 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40539.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41436.log.gz 2024-03-25T19:36:56.686 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-25T19:36:56.686 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40656.log 2024-03-25T19:36:56.686 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40812.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40539.log.gz 2024-03-25T19:36:56.686 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40344.log 2024-03-25T19:36:56.686 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40656.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40812.log.gz 2024-03-25T19:36:56.686 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42333.log 2024-03-25T19:36:56.686 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40344.log: 87.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41865.log 2024-03-25T19:36:56.686 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40656.log.gz 2024-03-25T19:36:56.686 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39953.log 2024-03-25T19:36:56.686 INFO:teuthology.orchestra.run.smithi083.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40344.log.gz 2024-03-25T19:36:56.687 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42216.log 2024-03-25T19:36:56.687 INFO:teuthology.orchestra.run.smithi083.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42333.log.gz 2024-03-25T19:36:56.687 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41865.log: /var/log/ceph/ceph-client.admin.39953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42177.log 2024-03-25T19:36:56.687 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42216.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41865.log.gz 2024-03-25T19:36:56.687 INFO:teuthology.orchestra.run.smithi083.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39953.log.gz 2024-03-25T19:36:56.687 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41982.log 2024-03-25T19:36:56.687 INFO:teuthology.orchestra.run.smithi083.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.42216.log.gz 2024-03-25T19:36:56.687 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42177.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-25T19:36:56.687 INFO:teuthology.orchestra.run.smithi083.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.42177.log.gz 2024-03-25T19:36:56.687 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40461.log 2024-03-25T19:36:56.687 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41826.log 2024-03-25T19:36:56.687 INFO:teuthology.orchestra.run.smithi083.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41982.log.gz 2024-03-25T19:36:56.687 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40461.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T19:36:56.687 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41826.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40461.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37026.log 2024-03-25T19:36:56.687 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T19:36:56.688 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41826.log.gz 2024-03-25T19:36:56.688 INFO:teuthology.orchestra.run.smithi083.stderr: --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-25T19:36:56.688 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.37026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42583.log 2024-03-25T19:36:56.688 INFO:teuthology.orchestra.run.smithi083.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.37026.log.gz 2024-03-25T19:36:56.688 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-osd.2.log: 92.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T19:36:56.688 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39719.log 2024-03-25T19:36:56.688 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42778.log 2024-03-25T19:36:56.688 INFO:teuthology.orchestra.run.smithi083.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42583.log.gz 2024-03-25T19:36:56.688 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.39719.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39719.log.gz 2024-03-25T19:36:56.688 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40617.log 2024-03-25T19:36:56.688 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42255.log 2024-03-25T19:36:56.688 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40617.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.42778.log.gz 2024-03-25T19:36:56.688 INFO:teuthology.orchestra.run.smithi083.stderr: 93.7% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-25T19:36:56.688 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42542.log 2024-03-25T19:36:56.689 INFO:teuthology.orchestra.run.smithi083.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40617.log.gz 2024-03-25T19:36:56.689 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40110.log 2024-03-25T19:36:56.689 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39914.log 2024-03-25T19:36:56.689 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42542.log: /var/log/ceph/ceph-client.admin.40110.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42255.log.gz 2024-03-25T19:36:56.689 INFO:teuthology.orchestra.run.smithi083.stderr: 87.0%gzip -5 --verbose 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40110.log.gz 2024-03-25T19:36:56.689 INFO:teuthology.orchestra.run.smithi083.stderr: -- /var/log/ceph/ceph-client.admin.39836.log 2024-03-25T19:36:56.689 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42542.log.gz/var/log/ceph/ceph-client.admin.39914.log: 2024-03-25T19:36:56.689 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42622.log 2024-03-25T19:36:56.689 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.39836.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39914.log.gz 2024-03-25T19:36:56.689 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-25T19:36:56.689 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42622.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39836.log.gz 2024-03-25T19:36:56.689 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-mon.c.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.42622.log.gz 2024-03-25T19:36:56.707 INFO:teuthology.orchestra.run.smithi103.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-25T19:36:57.516 INFO:teuthology.orchestra.run.smithi103.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-25T19:36:57.573 INFO:teuthology.orchestra.run.smithi103.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-25T19:36:57.805 INFO:teuthology.orchestra.run.smithi103.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-25T19:36:57.858 INFO:teuthology.orchestra.run.smithi083.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-25T19:36:57.865 INFO:teuthology.orchestra.run.smithi103.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-25T19:36:57.875 INFO:teuthology.orchestra.run.smithi083.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-25T19:36:57.882 INFO:teuthology.orchestra.run.smithi083.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-25T19:36:57.883 INFO:teuthology.orchestra.run.smithi083.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-25T19:37:01.505 INFO:teuthology.orchestra.run.smithi083.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-25T19:37:01.612 INFO:teuthology.orchestra.run.smithi103.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-25T19:37:01.614 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-25T19:37:01.614 INFO:teuthology.orchestra.run.smithi103.stderr:real 0m4.976s 2024-03-25T19:37:01.614 INFO:teuthology.orchestra.run.smithi103.stderr:user 0m8.877s 2024-03-25T19:37:01.614 INFO:teuthology.orchestra.run.smithi103.stderr:sys 0m0.305s 2024-03-25T19:37:01.749 INFO:teuthology.orchestra.run.smithi083.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-25T19:37:01.752 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T19:37:01.752 INFO:teuthology.orchestra.run.smithi083.stderr:real 0m5.140s 2024-03-25T19:37:01.752 INFO:teuthology.orchestra.run.smithi083.stderr:user 0m14.431s 2024-03-25T19:37:01.752 INFO:teuthology.orchestra.run.smithi083.stderr:sys 0m0.566s 2024-03-25T19:37:01.752 INFO:tasks.ceph:Archiving logs... 2024-03-25T19:37:01.752 DEBUG:teuthology.misc:Transferring archived files from smithi083:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621304/remote/smithi083/log 2024-03-25T19:37:01.753 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T19:37:04.059 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621304/remote/smithi103/log 2024-03-25T19:37:04.060 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T19:37:05.569 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T19:37:05.772 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=b49bbdda1e904c019007819852d5be7c Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T19:37:05.777 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-25T19:37:05.786 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-25T19:37:05.794 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-25T19:37:05.795 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T19:37:05.808 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T19:37:05.887 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-25T19:37:05.896 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-25T19:37:05.896 DEBUG:teuthology.orchestra.run.smithi083:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T19:37:05.899 DEBUG:teuthology.orchestra.run.smithi103:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T19:37:05.912 INFO:teuthology.orchestra.run.smithi083.stderr:bash: line 1: ntpq: command not found 2024-03-25T19:37:05.914 INFO:teuthology.orchestra.run.smithi103.stderr:bash: line 1: ntpq: command not found 2024-03-25T19:37:05.917 INFO:teuthology.orchestra.run.smithi083.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T19:37:05.917 INFO:teuthology.orchestra.run.smithi083.stdout:=============================================================================== 2024-03-25T19:37:05.917 INFO:teuthology.orchestra.run.smithi083.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 3 -205us[ -208us] +/- 81ms 2024-03-25T19:37:05.917 INFO:teuthology.orchestra.run.smithi083.stdout:^* hv02.front.sepia.ceph.com 3 6 377 1 -313us[ -316us] +/- 51ms 2024-03-25T19:37:05.917 INFO:teuthology.orchestra.run.smithi083.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 62 +769us[ +765us] +/- 85ms 2024-03-25T19:37:05.917 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T19:37:05.919 INFO:teuthology.orchestra.run.smithi103.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T19:37:05.919 INFO:teuthology.orchestra.run.smithi103.stdout:=============================================================================== 2024-03-25T19:37:05.919 INFO:teuthology.orchestra.run.smithi103.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 62 -202us[ -193us] +/- 80ms 2024-03-25T19:37:05.919 INFO:teuthology.orchestra.run.smithi103.stdout:^* hv02.front.sepia.ceph.com 3 6 377 3 -300us[ -292us] +/- 51ms 2024-03-25T19:37:05.919 INFO:teuthology.orchestra.run.smithi103.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 2 +803us[ +803us] +/- 86ms 2024-03-25T19:37:05.919 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T19:37:05.920 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-25T19:37:05.928 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-25T19:37:05.929 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-25T19:37:05.937 DEBUG:teuthology.orchestra.run.smithi083:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T19:37:06.029 DEBUG:teuthology.orchestra.run.smithi103:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T19:37:06.101 DEBUG:teuthology.orchestra.run.smithi083:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T19:37:06.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:37:06.132 DEBUG:teuthology.orchestra.run.smithi103:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T19:37:06.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:37:06.163 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-25T19:37:06.173 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-25T19:37:06.181 INFO:teuthology.task.internal:Duration was 6734.628581 seconds 2024-03-25T19:37:06.181 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-25T19:37:06.189 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-25T19:37:06.189 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T19:37:06.192 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T19:37:06.233 INFO:teuthology.orchestra.run.smithi083.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T19:37:06.248 INFO:teuthology.orchestra.run.smithi103.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T19:37:06.534 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-25T19:37:06.534 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi083.front.sepia.ceph.com 2024-03-25T19:37:06.535 DEBUG:teuthology.orchestra.run.smithi083:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T19:37:06.587 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi103.front.sepia.ceph.com 2024-03-25T19:37:06.587 DEBUG:teuthology.orchestra.run.smithi103:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T19:37:06.634 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-25T19:37:06.634 DEBUG:teuthology.orchestra.run.smithi083:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T19:37:06.636 DEBUG:teuthology.orchestra.run.smithi103:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T19:37:06.665 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-25T19:37:06.665 DEBUG:teuthology.orchestra.run.smithi083:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T19:37:06.708 DEBUG:teuthology.orchestra.run.smithi103:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T19:37:06.811 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-25T19:37:06.822 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-25T19:37:06.822 DEBUG:teuthology.orchestra.run.smithi083:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T19:37:06.848 DEBUG:teuthology.orchestra.run.smithi103:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T19:37:06.875 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-25T19:37:06.897 DEBUG:teuthology.orchestra.run.smithi083:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:37:06.900 DEBUG:teuthology.orchestra.run.smithi103:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:37:06.923 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern = core 2024-03-25T19:37:06.941 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern = core 2024-03-25T19:37:06.963 DEBUG:teuthology.orchestra.run.smithi083:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:37:07.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:37:07.017 DEBUG:teuthology.orchestra.run.smithi103:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:37:07.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:37:07.031 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-25T19:37:07.041 INFO:teuthology.task.internal:Transferring archived files... 2024-03-25T19:37:07.042 DEBUG:teuthology.misc:Transferring archived files from smithi083:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621304/remote/smithi083 2024-03-25T19:37:07.042 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T19:37:07.092 DEBUG:teuthology.misc:Transferring archived files from smithi103:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621304/remote/smithi103 2024-03-25T19:37:07.093 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T19:37:07.127 INFO:teuthology.task.internal:Removing archive directory... 2024-03-25T19:37:07.127 DEBUG:teuthology.orchestra.run.smithi083:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T19:37:07.129 DEBUG:teuthology.orchestra.run.smithi103:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T19:37:07.177 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-25T19:37:07.188 INFO:teuthology.task.internal:Not uploading archives. 2024-03-25T19:37:07.188 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-25T19:37:07.197 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-25T19:37:07.197 DEBUG:teuthology.orchestra.run.smithi083:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T19:37:07.200 DEBUG:teuthology.orchestra.run.smithi103:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T19:37:07.211 INFO:teuthology.orchestra.run.smithi083.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 19:37 /home/ubuntu/cephtest 2024-03-25T19:37:07.231 INFO:teuthology.orchestra.run.smithi103.stdout: 266042 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 19:37 /home/ubuntu/cephtest 2024-03-25T19:37:07.232 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-25T19:37:07.242 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-25T19:37:07.260 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T19:37:07.288 DEBUG:teuthology.nuke:shortname: smithi083 2024-03-25T19:37:07.288 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T19:37:07.313 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi083.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621304', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 17:35:18.260584', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOcxMrfnS1278Hh4wj+koiCBiAB8jKgK0bfQL+LG0aENfyqJ17ESta7jqnj6tNr7rTpGvziLwO1VCqfv+Nq7aD8='} 2024-03-25T19:37:07.316 DEBUG:teuthology.nuke:shortname: smithi103 2024-03-25T19:37:07.317 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T19:37:07.334 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi103.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621304', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 17:35:18.262282', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMS12b5fIyshgoPfOLpw9fDz704LM+m9FF/4ofwZhCYoyPVNHdYpMCoCpEbltvNc7n3eayx8YCzLHJb5tUCrrfs='} 2024-03-25T19:37:07.361 INFO:teuthology.orchestra.console.smithi083:Power off 2024-03-25T19:37:07.362 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T19:37:07.377 INFO:teuthology.orchestra.console.smithi103:Power off 2024-03-25T19:37:07.378 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T19:37:07.386 DEBUG:teuthology.orchestra.console.smithi083:power off output: Chassis Power Control: Down/Off 2024-03-25T19:37:07.387 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:37:07.399 DEBUG:teuthology.orchestra.console.smithi103:power off output: Chassis Power Control: Down/Off 2024-03-25T19:37:07.399 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:37:07.410 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is on 2024-03-25T19:37:07.422 DEBUG:teuthology.orchestra.console.smithi103:check power output: Chassis Power is on 2024-03-25T19:37:11.412 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:37:11.423 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:37:11.523 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is on 2024-03-25T19:37:11.533 DEBUG:teuthology.orchestra.console.smithi103:check power output: Chassis Power is on 2024-03-25T19:37:15.528 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:37:15.536 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:37:15.639 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is on 2024-03-25T19:37:15.645 DEBUG:teuthology.orchestra.console.smithi103:check power output: Chassis Power is off 2024-03-25T19:37:15.745 INFO:teuthology.orchestra.console.smithi103:Power off completed 2024-03-25T19:37:19.643 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:37:19.753 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is off 2024-03-25T19:37:19.853 INFO:teuthology.orchestra.console.smithi083:Power off completed 2024-03-25T19:37:19.958 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} duration: 6734.6285808086395 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=b49bbdda1e904c019007819852d5be7c status: fail success: false 2024-03-25T19:37:19.958 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T19:37:20.039 INFO:teuthology.run:FAIL